$ guides / proxysql / proxysql-access-denied-for-user ▌

Operations Guides

ProxySQL error 1045 Access denied for user: credential rotation not propagated

Error 1045 is MySQL’s “Access denied for user.” In a ProxySQL environment it can surface at three independent authentication boundaries, and fixing the wrong one wastes the incident window. The typical trigger is a password rotation applied to one layer (backend MySQL, application config, or ProxySQL itself) without propagating it through ProxySQL’s three-layer configuration model.

The error string carries an immediate clue. (using password: YES) means a password was sent but did not match. (using password: NO) means no password was sent at all, which can indicate a client misconfiguration or, rarely, a version-specific bug. Most credential rotation incidents produce (using password: YES).

The authentication cascade is recognizable in the metrics: Access_Denied_Wrong_Password climbs while the Questions rate drops and Client_Connections_created stays above zero. Clients are actively trying to connect, failing at the proxy, and never reaching the query processor. The correct fix depends on which layer is rejecting the credential.

What this means

ProxySQL authenticates at three independent boundaries. Each uses its own credential store and its own propagation path.

Frontend (client to ProxySQL, port 6033). Clients authenticate against ProxySQL’s mysql_users table. ProxySQL validates the password the client sends against the password column for the matching username. If the password in mysql_users was updated but the application still sends the old password, every new connection fails with 1045.

Backend (ProxySQL to MySQL). When ProxySQL borrows a backend connection from the pool, it authenticates to the MySQL server using the same password from mysql_users. If the password was rotated on the backend but not updated in mysql_users, ProxySQL cannot establish backend connections. ConnERR rises, backends accumulate failures, and after the configured shun_on_failures threshold they are SHUNNED.

Monitor (ProxySQL monitor to MySQL). The monitor module uses a separate credential pair: mysql-monitor_username and mysql-monitor_password, configured as global variables, not in mysql_users. If these become stale after a backend rotation, health checks fail. ProxySQL cannot verify backend health and shuns backends that are actually fine. Data queries using the correct mysql_users password would succeed if they could reach the backend, but the monitor has already pulled the backend from rotation.

The three-layer configuration model adds another dimension. Changes via the admin interface land in MEMORY. They are not active until LOAD ... TO RUNTIME, and not durable across restarts until SAVE ... TO DISK. A credential update committed to MEMORY but never loaded to RUNTIME has no effect. One loaded to RUNTIME but never saved to DISK works until the next restart, then silently reverts.

flowchart TD
    A["Error 1045 reported"] --> B{"Access_Denied_Wrong_Password rising?"}
    B -->|"Yes"| C["Frontend: mysql_users mismatch"]
    B -->|"No"| D{"ConnERR rising on backends?"}
    D -->|"Yes"| E["Backend: mysql_users vs backend grant"]
    D -->|"No"| F{"Monitor_connect_check_ERR rising?"}
    F -->|"Yes"| G["Monitor: mysql-monitor_password stale"]
    F -->|"No"| H["Check other causes"]

Common causes

Cause	What it looks like	First thing to check
Password rotated on backend MySQL, not updated in ProxySQL `mysql_users`	`ConnERR` rising on all backends; backends SHUNNED; `stats_mysql_errors` shows errno 1045 from backend hostgroups	Compare `mysql_users.password` with the backend MySQL grant
Password updated in `mysql_users` but applications still send old password	`Access_Denied_Wrong_Password` rising; `Client_Connections_aborted` spiking; `Questions` dropping	Verify what password the application is configured to send
Monitor password stale (`mysql-monitor_password` does not match backend)	`MySQL_Monitor_connect_check_ERR` rising; backends SHUNNED despite being reachable directly; data queries may still work if `mysql_users` password is correct	Check `mysql-monitor_password` in `global_variables` against the backend grant
`LOAD MYSQL USERS TO RUNTIME` not executed after update	`mysql_users` (MEMORY) differs from `runtime_mysql_users` (RUNTIME); changes have no effect	Compare password columns in both tables
`SAVE MYSQL USERS TO DISK` not executed after update	Changes work at runtime but revert after ProxySQL restart	Check the on-disk database or re-apply and save
ProxySQL cluster node not synced	Error appears from one ProxySQL instance but not others	Check `stats_proxysql_servers_checksums` for `mysql_users` divergence

Quick checks

All commands connect to the admin interface on port 6032. Replace default credentials (admin/admin) with your actual admin user and password.

# Global auth failure counter and impact signals
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global
      WHERE Variable_Name IN ('Access_Denied_Wrong_Password',
                              'Client_Connections_aborted',
                              'Client_Connections_created',
                              'Questions',
                              'ProxySQL_Uptime');"

# Per-backend connection errors (ConnERR increasing = backend auth failing)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status,
             ConnUsed, ConnFree, ConnOK, ConnERR
      FROM stats_mysql_connection_pool;"

# stats_mysql_errors for errno 1045 (backend auth failures)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM stats_mysql_errors
      WHERE errno = 1045 ORDER BY last_seen DESC LIMIT 20;"

# Compare mysql_users (MEMORY) vs runtime_mysql_users (RUNTIME) for drift
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT username, password FROM mysql_users WHERE username='app_user';
      SELECT username, password FROM runtime_mysql_users WHERE username='app_user';"

# Monitor credentials (MEMORY vs RUNTIME)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM global_variables
      WHERE variable_name IN ('mysql-monitor_username','mysql-monitor_password');
      SELECT variable_name, variable_value FROM runtime_global_variables
      WHERE variable_name IN ('mysql-monitor_username','mysql-monitor_password');"

# Monitor check failure rates
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global
      WHERE Variable_Name LIKE 'MySQL_Monitor_connect_check%';"

# ProxySQL cluster checksums (if clustered)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM stats_proxysql_servers_checksums;"

How to diagnose it

Identify the failing boundary. If Access_Denied_Wrong_Password is rising and Client_Connections_aborted is spiking while Questions drops, the problem is frontend: clients cannot authenticate to ProxySQL. If ConnERR is rising on backends and stats_mysql_errors shows errno 1045 from backend hostgroups, the problem is backend: ProxySQL cannot authenticate to MySQL. If MySQL_Monitor_connect_check_ERR is rising and backends are being SHUNNED, the monitor credentials are stale.
Check for config layer drift. Compare mysql_users (MEMORY) with runtime_mysql_users (RUNTIME). If the password differs, the change was never loaded to runtime. This is the most common cause of “I updated the password but nothing changed.”
Verify the password against the backend grant. Connect directly to a backend MySQL server (bypassing ProxySQL) using the password from mysql_users. If it fails, the password in mysql_users is wrong. If it succeeds but ProxySQL still reports ConnERR, confirm that runtime_mysql_users matches mysql_users.
Check monitor credentials independently. Even if mysql_users is correct, stale mysql-monitor_password causes backends to be SHUNNED. Connect directly to the backend using mysql-monitor_username and mysql-monitor_password to verify they work.
Check cluster sync lag. In a ProxySQL cluster, stats_proxysql_servers_checksums shows whether all nodes have the same mysql_users revision. A checksum mismatch means one node is serving stale credentials.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`Access_Denied_Wrong_Password`	Tracks frontend auth failures from wrong passwords	Sustained increase with `Questions` dropping
`Client_Connections_aborted`	Clients rejected after connecting	Rising rate correlated with auth failures
`Questions` rate	Query throughput drops when auth blocks clients from the query processor	Sudden sustained drop below baseline
`ConnERR` per backend	Backend connection failures including auth rejection	Rising on all backends in a hostgroup simultaneously
`stats_mysql_errors` (errno 1045)	Per-hostgroup, per-host error breakdown	Error 1045 rows from backend hostgroups
`MySQL_Monitor_connect_check_ERR`	Monitor cannot connect to backends	Sustained increase signals stale monitor credentials
Backend status (ONLINE / SHUNNED)	Monitor shuns backends it cannot health-check	Backends SHUNNED despite being reachable directly

Fixes

The commands below modify ProxySQL configuration. Each LOAD ... TO RUNTIME activates changes immediately. Each SAVE ... TO DISK writes to the persistent SQLite database. Run them in order and verify between steps.

Frontend credentials: application sends wrong password to ProxySQL

If the password in mysql_users is correct (matches the backend) but applications are sending the old password, the fix is on the application side. If you rotated the password in mysql_users and need to push it to runtime:

UPDATE mysql_users SET password='new_password' WHERE username='app_user';
LOAD MYSQL USERS TO RUNTIME;
SAVE MYSQL USERS TO DISK;

To revert temporarily while applications catch up:

UPDATE mysql_users SET password='old_password' WHERE username='app_user';
LOAD MYSQL USERS TO RUNTIME;
SAVE MYSQL USERS TO DISK;

Backend credentials: ProxySQL sends wrong password to MySQL

Same fix path. The mysql_users.password column serves both layers: it validates client connections (frontend) and authenticates ProxySQL to the backend. Updating it fixes both simultaneously.

For zero-downtime rotation on ProxySQL v3.0+, consider dual passwords via the attributes column in mysql_users, which allows an additional_password to coexist with the primary.

Monitor credentials

Monitor credentials are global variables, not part of mysql_users. They require a separate load and save cycle:

UPDATE global_variables SET variable_value='new_password'
  WHERE variable_name='mysql-monitor_password';
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

Config layer drift

If mysql_users (MEMORY) differs from runtime_mysql_users (RUNTIME), the change was never loaded:

LOAD MYSQL USERS TO RUNTIME;

If loaded to RUNTIME but never saved to DISK, it will revert on restart:

SAVE MYSQL USERS TO DISK;

ProxySQL cluster sync

In a cluster, credential changes must propagate to all nodes. After applying the change on one node, verify convergence via stats_proxysql_servers_checksums. If checksums do not converge, manually load and save on the lagging node.

Version-specific gotchas

admin-hash_passwords (deprecated in v2.6.0). Before v2.6.0, when admin-hash_passwords=true (the default), ProxySQL automatically hashed cleartext passwords in mysql_users during LOAD MYSQL USERS TO RUNTIME. The mysql_users table retained cleartext while runtime_mysql_users held hashes. Comparing the two tables would show different values by design, not drift. Since v2.6.0 this variable is deprecated.

Password storage format. ProxySQL accepts cleartext passwords and hashed formats. MySQL native password hashes use the * prefix; caching_sha2_password hashes use the $A$ prefix. If the backend uses caching_sha2_password and your ProxySQL version stores the password in an incompatible format, authentication fails.

Prevention

Credential rotation runbook. Every password change must touch three places: mysql_users (or mysql-monitor_password for monitor), LOAD ... TO RUNTIME, and SAVE ... TO DISK. Missing any step produces either no effect or a silent reversion on restart.
Monitor config layer drift. Periodically compare mysql_users with runtime_mysql_users, and global_variables with runtime_global_variables for monitor credentials. No built-in metric detects this; it requires an explicit query comparison.
Rotate monitor credentials with backend credentials. The monitor uses a separate password. If you rotate the backend password, update mysql-monitor_password in the same change window.
Use dual passwords for zero-downtime rotation. MySQL 8.0+ supports retaining both a primary and a secondary password during rotation. ProxySQL v3.0+ supports an analogous feature via the attributes column. This eliminates the window where either the old or new password fails.
Verify cluster convergence. After any credential change in a clustered deployment, check stats_proxysql_servers_checksums to confirm all nodes have the same mysql_users revision.
Gate the auth cascade alert. Page only when Access_Denied_Wrong_Password is sustained AND Questions is dropping AND Client_Connections_created > 0 AND ProxySQL_Uptime > 600. This excludes cold starts, idle instances, and brief blips during rotation windows.

How Netdata helps

Per-second Access_Denied_Wrong_Password collection shows the exact moment credential propagation breaks or succeeds during a rotation window.
Correlating Access_Denied_Wrong_Password with Questions and Client_Connections_aborted in a single view distinguishes a frontend auth cascade from a backend connectivity problem without running separate admin queries.
Backend ConnERR per host reveals whether the problem is credential-wide (all backends failing) or isolated to one host.
Monitor check failure rates (MySQL_Monitor_connect_check_ERR) surface stale mysql-monitor_password before backends get SHUNNED.
Backend status changes (ONLINE to SHUNNED) correlated with monitor check errors distinguish “backend is unhealthy” from “ProxySQL cannot authenticate to a healthy backend.”