$ guides / proxysql / proxysql-monitor-password-invalid ▌

Operations Guides

ProxySQL monitor password invalid: healthy backends shunned because health checks fail

Backends are going SHUNNED, but direct connections to the MySQL servers work fine. The servers are up, responsive, and serving queries normally. Yet ProxySQL has pulled them from rotation.

The backends are not the problem. The Monitor module is.

When mysql-monitor_password is wrong, expired, or out of sync, every monitor connect and ping check fails with an authentication error. ProxySQL interprets these failures as backend health failures. After mysql-monitor_ping_max_failures (default: 3) consecutive ping failures, it SHUNS the backend and closes its connections. The backend is healthy, but ProxySQL cannot verify that, so it stops sending traffic.

What happens

The Monitor module authenticates to each backend with a dedicated credential pair: mysql-monitor_username (default: monitor) and mysql-monitor_password (default: monitor). These are separate from application credentials in mysql_users. The monitor username cannot be used in mysql_users, and mysql_users credentials are not used for health checks.

When the monitor password is wrong on a backend:

Connect and ping checks fail with access denied.
MySQL_Monitor_connect_check_ERR and MySQL_Monitor_ping_check_ERR climb steadily.
After mysql-monitor_ping_max_failures consecutive ping failures, ProxySQL SHUNS the backend.
Read-only checks also fail since they use the same credential.

The data-plane credentials in mysql_users may be correct. Application queries would succeed if ProxySQL would route them. But the health check cannot authenticate, so ProxySQL treats the backend as failed.

This looks exactly like a real backend outage in the metrics: backends going SHUNNED, monitor check errors rising, traffic shifting. The root cause is a single credential that is easy to miss during password rotations, server migrations, or ProxySQL restarts with stale disk configuration.

flowchart TD
    A["Password rotated on MySQL backend"] --> B["mysql-monitor_password in ProxySQL is stale"]
    B --> C["Monitor connect check: Access denied"]
    C --> D["MySQL_Monitor_connect_check_ERR climbs"]
    C --> E["Monitor ping check: Access denied"]
    E --> F["MySQL_Monitor_ping_check_ERR climbs"]
    D --> G["Failures reach ping_max_failures threshold"]
    F --> G
    G --> H["ProxySQL SHUNS the backend"]
    H --> I["Backend healthy but removed from rotation"]

Common causes

Cause	What it looks like	First thing to check
Password rotated on MySQL backends but not in ProxySQL	All backends in a hostgroup start failing monitor checks simultaneously after a credential rotation	Query `global_variables` for the current monitor password and test it directly against a backend
Password SET but never LOADed to RUNTIME	Password was updated via the admin interface but monitor checks still fail	Run `LOAD MYSQL VARIABLES TO RUNTIME` and observe whether checks recover
Password loaded to RUNTIME but never SAVEd to DISK	Monitor checks worked until the last ProxySQL restart, then all backends went SHUNNED	Check whether a recent restart coincides with the failure onset
Monitor user uses caching_sha2_password on MySQL 8	Monitor check errors persist even though the password is correct	Check the auth plugin for the monitor user on the backend
Monitor user dropped or privileges revoked on backend	All check types fail for specific backends while others remain healthy	Verify the monitor user exists with USAGE privilege on each backend

Diagnosis

All SQL commands run against the ProxySQL admin interface on port 6032.

1. Confirm backends are healthy independently

Connect to the MySQL server directly, bypassing the proxy, and run a simple query. If it works, the backend is fine and the problem is in ProxySQL’s health checking.

mysql -u <app_user> -p'<app_password>' -h <backend_host> -P <backend_port> -e "SELECT 1;"

2. Check which backends are SHUNNED

SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree, ConnERR
FROM stats_mysql_connection_pool;

If ConnERR is zero while backends are SHUNNED, the problem is monitor-only, not data-plane.

3. Check monitor check error counters

SELECT Variable_Name, Variable_Value FROM stats_mysql_global
WHERE Variable_Name LIKE 'MySQL_Monitor_%check%';

If MySQL_Monitor_connect_check_ERR and MySQL_Monitor_ping_check_ERR are climbing while OK counters are flat, the monitor module is consistently failing to authenticate.

4. Read the stored monitor credentials

SELECT variable_name, variable_value FROM global_variables
WHERE variable_name IN ('mysql-monitor_username','mysql-monitor_password');

5. Test the stored credential against a backend

mysql -u monitor -p'<monitor_password_from_step_4>' -h <backend_host> -P <backend_port> -e "SELECT 1;"

If this fails with “Access denied,” the credential itself is wrong. If it succeeds, the credential is correct but may not be loaded to RUNTIME.

6. Check the monitor log tables for error detail

SELECT * FROM monitor.mysql_server_ping_log
ORDER BY time_start_us DESC LIMIT 20;

Failed entries show the error detail. Authentication errors confirm the credential problem.

7. Check the backend MySQL error log

The error log path varies by distribution and configuration. Common locations: /var/log/mysql/error.log, /var/log/mysqld.log, or wherever log_error points.

grep -i "access denied" /var/log/mysql/error.log | grep -i monitor | tail -20

“Access denied for user” entries matching the monitor username confirm the backend is rejecting authentication.

8. Verify config layer consistency

If the password is correct in global_variables but checks still fail, the value may exist only in MEMORY and was never propagated to RUNTIME. See how ProxySQL actually works in production for the three-layer model.

9. Check for auth plugin mismatch

If the password is definitely correct but checks still fail on MySQL 8, check whether the monitor user uses caching_sha2_password:

-- On the MySQL backend
SELECT user, host, plugin FROM mysql.user WHERE user = 'monitor';

ProxySQL may not support this auth plugin for the monitor user. Change the monitor user to mysql_native_password on the backend.

Fixes

Update the monitor password

The primary fix. Update the credential, load it to RUNTIME, and persist to DISK:

SET mysql-monitor_password = '<new_password>';
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

After loading to RUNTIME, the monitor module uses the new password on the next check cycle. SHUNNED backends should recover automatically within one or two check intervals.

Verify recovery:

SELECT hostgroup, srv_host, srv_port, status FROM stats_mysql_connection_pool;

SELECT Variable_Name, Variable_Value FROM stats_mysql_global
WHERE Variable_Name LIKE 'MySQL_Monitor_%check%';

Fix config drift: password SET but never LOADed

The password is correct in global_variables but monitor checks still fail. The value exists only in MEMORY:

LOAD MYSQL VARIABLES TO RUNTIME;

This is the most common variant. An operator updates the password via SET, assumes it takes effect immediately, and walks away.

Fix config drift: password loaded to RUNTIME but never SAVEd to DISK

Monitor checks worked until the last ProxySQL restart, then all backends went SHUNNED. The restart loaded the old DISK value. Re-apply and complete all three steps:

SET mysql-monitor_password = '<correct_password>';
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

Fix auth plugin mismatch on MySQL 8

Change the monitor user’s auth plugin on the backend:

-- On the MySQL 8 backend
ALTER USER 'monitor'@'%' IDENTIFIED WITH mysql_native_password BY '<password>';

This only affects the monitor user’s authentication, not application users.

ProxySQL cluster propagation

In a ProxySQL cluster, variable changes propagate through the cluster sync mechanism. After updating the password on one node, verify convergence:

SELECT * FROM stats_proxysql_servers_checksums;

Checksums should match across all nodes after propagation completes.

Metrics and signals

Signal	Why it matters	Warning sign
`MySQL_Monitor_connect_check_ERR`	Failed connect checks	Steady climb with no corresponding backend fault
`MySQL_Monitor_ping_check_ERR`	Failed ping checks	Parallel climb with connect_check_ERR
`MySQL_Monitor_read_only_check_ERR`	Read-only checks also use the monitor credential	Climbs alongside connect and ping errors
Backend `status` in `stats_mysql_connection_pool`	Whether ProxySQL has pulled the backend from rotation	ONLINE to SHUNNED on independently healthy backends
`ConnERR` in `stats_mysql_connection_pool`	Data-plane connection errors	Zero while backends are SHUNNED: problem is monitor-only
`MySQL_Monitor_Workers`	Whether monitor threads are active	Zero means monitoring is disabled
`monitor.mysql_server_ping_log`	Per-check results with timestamps and errors	“Access denied” entries confirming auth failures

Prevention

Include the monitor credential in every password rotation runbook. The mysql-monitor_password is separate from mysql_users and easy to miss. Any procedure that rotates MySQL credentials must include the full SET, LOAD, SAVE sequence.

Alert on sustained increases in MySQL_Monitor_connect_check_ERR or MySQL_Monitor_ping_check_ERR. These counters start climbing before backends are shunned.

After any credential update, confirm the value is loaded to RUNTIME and saved to DISK.

Use mysql_native_password for the monitor user on MySQL 8 backends.

Create the monitor user with minimal privileges. Connect and ping checks need only USAGE. Add REPLICATION CLIENT only if you use replication lag monitoring.

How Netdata helps

Per-second collection of MySQL_Monitor_*_check_ERR counters lets you pinpoint when monitor checks started failing and correlate against credential rotation events or ProxySQL restarts.
Backend status changes (ONLINE to SHUNNED) are tracked per-backend with per-second granularity.
Correlating monitor check failures with zero data-plane ConnERR distinguishes a monitor credential problem from a genuine backend connectivity issue.
MySQL_Monitor_Workers visibility confirms the monitor module is running, ruling out a disabled or starved monitor.