$ guides / proxysql / proxysql-config-changes-not-applied ▌

Operations Guides

ProxySQL config changes not applied: the LOAD TO RUNTIME / SAVE TO DISK trap

You INSERT or UPDATE a row in mysql_servers, mysql_users, or mysql_query_rules through the admin interface. The SQL succeeds with no error. But traffic behavior does not change. Or it works fine for a day, then ProxySQL restarts and everything reverts.

The root cause is the three-layer configuration model. Every admin SQL change lands in MEMORY, the staging area. That change is not active until you explicitly run LOAD ... TO RUNTIME. And it does not survive a restart until you explicitly run SAVE ... TO DISK. No transition is automatic. No error fires if you skip a step.

Each configuration module has its own LOAD and SAVE commands. LOAD MYSQL USERS TO RUNTIME activates user changes. LOAD MYSQL SERVERS TO RUNTIME activates server changes. LOAD MYSQL QUERY RULES TO RUNTIME activates query rule changes. LOAD MYSQL VARIABLES TO RUNTIME activates variable changes. If you change three modules and only load one, the other two sit in MEMORY with no effect and no warning.

What this means

ProxySQL stores configuration across three layers:

MEMORY: The staging area, also called the main schema. When you run INSERT, UPDATE, or DELETE against mysql_servers, mysql_users, or mysql_query_rules, or SET on global variables, the change lands here. Changes in MEMORY have zero effect on live traffic.
RUNTIME: The active configuration governing connection routing, authentication, query processing, and backend health. The runtime_mysql_servers, runtime_mysql_users, and runtime_mysql_query_rules tables reflect what ProxySQL is actually doing right now. A change in MEMORY becomes active only when you run LOAD ... TO RUNTIME for the relevant module.
DISK: The persistent SQLite database (proxysql.db). On restart, ProxySQL loads configuration from DISK into RUNTIME. A change loaded to RUNTIME but never saved to DISK vanishes on the next restart.

flowchart TD
    A["Admin SQL change"] --> B["MEMORY layer
mysql_servers, mysql_users,
mysql_query_rules, variables"]
    B -->|"LOAD MODULE TO RUNTIME"| C["RUNTIME layer
runtime_mysql_servers,
runtime_mysql_users, etc."]
    C -->|"SAVE MODULE TO DISK"| D["DISK layer
proxysql.db SQLite"]
    D -->|"On restart: loads from disk"| C
    B -.-|"Forgot LOAD?"| E["Change not active
No error, no signal"]
    C -.-|"Forgot SAVE?"| F["Change works now
Lost after restart"]

The critical property: changes left in MEMORY produce no error. Changes loaded to RUNTIME but not saved to DISK work perfectly until a restart. Both failure modes are silent. There is no metric that says “you have uncommitted configuration.”

Common causes

Cause	What it looks like	First thing to check
Forgot `LOAD ... TO RUNTIME`	SQL change succeeded, behavior unchanged, no error returned	Compare `mysql_servers` vs `runtime_mysql_servers`
Forgot `SAVE ... TO DISK`	Change works after LOAD, reverts after ProxySQL restart	Check whether runtime config matches disk after restart
Partial module LOAD	Changed multiple modules, only loaded one. Some changes active, others not.	Check each `runtime_*` table individually
Wrong LOAD command	Changed `mysql_query_rules` but only ran `LOAD MYSQL USERS TO RUNTIME`	Match the LOAD command to the module you modified
`mysql-interfaces` change	Changed `mysql-interfaces`, loaded to runtime, listener did not update	`mysql-interfaces` requires SAVE TO DISK and restart; cannot take effect at runtime
Cluster node divergence	Change applied on one node, not synced to peers	Check `stats_proxysql_servers_checksums` for mismatched checksums

Quick checks

All queries below are read-only and safe against a production ProxySQL admin interface.

-- Connect: mysql -u admin -padmin -h 127.0.0.1 -P 6032
-- Default credentials admin:admin should be changed in production

-- Compare MEMORY vs RUNTIME for servers
SELECT hostgroup_id, hostname, port, status, max_connections FROM mysql_servers;
SELECT hostgroup_id, hostname, port, status, max_connections FROM runtime_mysql_servers;

-- Compare MEMORY vs RUNTIME for users
SELECT username, default_hostgroup, active FROM mysql_users;
SELECT username, default_hostgroup, active FROM runtime_mysql_users;

-- Compare MEMORY vs RUNTIME for query rules
SELECT rule_id, match_digest, destination_hostgroup, active, apply
  FROM mysql_query_rules ORDER BY rule_id;
SELECT rule_id, match_digest, destination_hostgroup, active, apply
  FROM runtime_mysql_query_rules ORDER BY rule_id;

-- Did ProxySQL recently restart? (forgot SAVE TO DISK indicator)
SELECT Variable_Name, Variable_Value FROM stats_mysql_global
  WHERE Variable_Name = 'ProxySQL_Uptime';

-- Did runtime server config actually change?
SELECT Variable_Name, Variable_Value FROM stats_mysql_global
  WHERE Variable_Name = 'Servers_table_version';

-- Cluster: are all peers in sync?
SELECT * FROM stats_proxysql_servers_checksums;

If mysql_servers has rows that runtime_mysql_servers does not (or has different values), you have uncommitted changes. The same logic applies to users and query rules.

For faster comparison without full row sets, ProxySQL supports CHECKSUM commands:

-- Compare MEMORY vs RUNTIME for servers
CHECKSUM MEMORY MYSQL SERVERS;
CHECKSUM RUNTIME MYSQL SERVERS;

-- Compare DISK vs MEMORY for servers
CHECKSUM DISK MYSQL SERVERS;
CHECKSUM MEMORY MYSQL SERVERS;

If the checksum values match, the layers are consistent. If they differ, you have drift.

How to diagnose it

Identify which module you changed. Was it mysql_servers, mysql_users, mysql_query_rules, or a global variable? Each requires its own LOAD command.
Compare MEMORY vs RUNTIME for that module. Query both the staging table and the runtime table. If they differ, the change was never loaded.
Check ProxySQL_Uptime. If uptime is low (recently restarted) and the change was working before the restart, the change was likely never saved to disk. On restart, ProxySQL loads from DISK, discarding anything that was only in RUNTIME.
Check stats_proxysql_servers_checksums (cluster only). If you run multiple ProxySQL instances, mismatched checksums between peers mean one node has different config. Clients hitting different ProxySQL instances will experience different routing behavior.
Verify the LOAD command matches the module. LOAD MYSQL SERVERS TO RUNTIME covers mysql_servers (and mysql_replication_hostgroups). LOAD MYSQL USERS TO RUNTIME covers mysql_users. LOAD MYSQL QUERY RULES TO RUNTIME covers mysql_query_rules. LOAD MYSQL VARIABLES TO RUNTIME covers global variables. Running the wrong LOAD leaves your actual change sitting in MEMORY.
Check for special-case variables. If you changed mysql-interfaces, a standard LOAD MYSQL VARIABLES TO RUNTIME will not apply the change. You must SAVE MYSQL VARIABLES TO DISK and restart ProxySQL.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`Servers_table_version`	Increments when server config changes at runtime	No increment after your change means LOAD did not happen. Unexpected increment means someone else changed config.
`ProxySQL_Uptime`	Time since last restart	Low uptime combined with reverted config means forgot SAVE TO DISK.
`stats_proxysql_servers_checksums`	Config checksums per module across cluster peers	Mismatched checksums mean nodes are routing differently.
`mysql_servers` vs `runtime_mysql_servers`	Direct comparison of staging vs active config	Row count or content differences mean uncommitted changes.
`Access_Denied_Wrong_Password`	Authentication failures from credential mismatch	Spike after a user change means user config was not loaded or wrong password was saved.
Backend `ConnERR`	Connection errors to backend servers	Spike after server change means new config was not loaded, or has incorrect values.

Fixes

Change in MEMORY not loaded to RUNTIME

Run the appropriate LOAD command for each module you changed:

LOAD MYSQL SERVERS TO RUNTIME;       -- servers + replication hostgroups
LOAD MYSQL USERS TO RUNTIME;         -- user credentials and routing
LOAD MYSQL QUERY RULES TO RUNTIME;   -- query routing rules
LOAD MYSQL VARIABLES TO RUNTIME;     -- global mysql_* variables
LOAD ADMIN VARIABLES TO RUNTIME;     -- admin_* variables

Verify by comparing the MEMORY and RUNTIME tables afterward. They should match.

Change in RUNTIME not saved to DISK

Run the appropriate SAVE command immediately:

SAVE MYSQL SERVERS TO DISK;
SAVE MYSQL USERS TO DISK;
SAVE MYSQL QUERY RULES TO DISK;
SAVE MYSQL VARIABLES TO DISK;
SAVE ADMIN VARIABLES TO DISK;

If you discover this after a restart already happened, you must re-apply the change from scratch: modify the table, LOAD TO RUNTIME, then SAVE TO DISK. The previous runtime state is gone.

Partial module loading

If you changed multiple modules, enumerate which ones and LOAD each individually. There is no single “load everything” command. After loading each module, SAVE each to DISK. The modules that require independent handling are:

MYSQL SERVERS (also covers mysql_replication_hostgroups)
MYSQL USERS
MYSQL QUERY RULES
MYSQL VARIABLES
ADMIN VARIABLES

Cluster divergence

If one node has different config from its peers, load and save on the authoritative node:

LOAD MYSQL SERVERS TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

ProxySQL cluster sync is eventually consistent. After loading and saving on the node with the correct config, changes propagate to peers. Verify convergence with stats_proxysql_servers_checksums.

Note: in ProxySQL Cluster mode, admin-cluster_mysql_users_save_to_disk defaults to true. After a remote cluster sync and LOAD TO RUNTIME, synced users may be automatically persisted to disk even without an explicit SAVE. Similar variables exist for mysql_servers, mysql_query_rules, and proxysql_servers. If you depend on runtime-only changes staying out of disk, verify these cluster save settings.

Prevention

Standardize the three-command workflow. Every config change follows the same pattern: modify (INSERT/UPDATE/SET), LOAD TO RUNTIME, SAVE TO DISK.

Create a post-change verification step. After LOAD, compare the MEMORY and RUNTIME tables for the module you changed. This catches the “I thought I loaded it” mistake before it causes an incident.

Maintain a per-module LOAD/SAVE reference. Put it in the runbook or as comments in your config management tool:

-- Servers + replication hostgroups
LOAD MYSQL SERVERS TO RUNTIME;  SAVE MYSQL SERVERS TO DISK;
-- User credentials
LOAD MYSQL USERS TO RUNTIME;    SAVE MYSQL USERS TO DISK;
-- Query routing rules
LOAD MYSQL QUERY RULES TO RUNTIME;  SAVE MYSQL QUERY RULES TO DISK;
-- Global mysql_* variables
LOAD MYSQL VARIABLES TO RUNTIME;  SAVE MYSQL VARIABLES TO DISK;
-- Admin variables
LOAD ADMIN VARIABLES TO RUNTIME;  SAVE ADMIN VARIABLES TO DISK;

Treat restarts as config-validation events. Before any planned restart, verify that RUNTIME matches DISK for all modules. After any restart, verify that the active config matches what you expect.

Monitor config divergence actively. There is no built-in metric for this. Query the tables directly or use CHECKSUM commands. In a cluster, monitor stats_proxysql_servers_checksums for peer divergence and alert on sustained mismatches.

Audit config changes after incidents. When an operator makes a quick fix under pressure, the temptation to skip SAVE TO DISK is high. The fix works immediately after LOAD TO RUNTIME, the incident resolves, and everyone moves on. Days later, a restart reverts the fix and the incident recurs. Require a post-incident config audit that verifies all three layers are consistent.

How Netdata helps

Netdata’s ProxySQL collector surfaces signals that help detect and correlate config drift:

ProxySQL_Uptime tracking: A restart event that correlates with a behavioral change (backend status shifts, routing changes, auth failure patterns) is the signature of a missing SAVE TO DISK. Per-second uptime metrics make the restart timestamp precise enough to confirm causation.
Backend status changes: If a server config change was not loaded to RUNTIME, stats_mysql_connection_pool still reflects the old routing. Backend status and connection distribution metrics make the mismatch visible when compared against the intended config.
Authentication failure rates: Access_Denied_Wrong_Password spiking after a credential change that was saved to disk but not loaded to runtime (or loaded with the wrong value) is a direct signal of config layer divergence.
Cluster checksum monitoring: For ProxySQL Cluster deployments, stats_proxysql_servers_checksums reveals when nodes have diverged. Per-second collection makes transient divergence during config propagation distinguishable from persistent split-brain.
Connection error correlation: ConnERR spikes after a server config change suggest the new config was not loaded properly, or was loaded with incorrect values that ProxySQL accepted in MEMORY but that fail against the actual backend.
Servers_table_version tracking: This counter increments when runtime server config changes. If you made an admin change but the counter did not increment, the LOAD TO RUNTIME did not happen.