$ guides / proxysql / proxysql-config-lost-after-restart ▌

Operations Guides

ProxySQL config lost after restart: runtime never saved to disk

ProxySQL separates live configuration from persistent configuration. If you apply a change to RUNTIME but never run SAVE ... TO DISK, that change survives only until the next restart. When ProxySQL restarts, it loads from proxysql.db on disk, and everything unsaved is gone.

The three-layer model means making a change live and making it durable are two separate, explicit steps. There is no single “save everything” command. If you forget any module’s SAVE ... TO DISK, that module reverts on the next restart.

The inverse failure is equally dangerous: starting ProxySQL with --reload or --initial reads the config file (/etc/proxysql.cnf) and overwrites the SQLite database, discarding everything that was saved via the admin interface. If the config file was never updated to match your live configuration, you lose all of it.

What this means

ProxySQL stores configuration in three layers:

MEMORY: The in-memory staging area. When you run UPDATE mysql_servers SET ... through the admin interface (port 6032), the change lands here. It is not active yet.
RUNTIME: The live configuration used by worker threads. LOAD <MODULE> TO RUNTIME copies from MEMORY to RUNTIME. This is what actually serves traffic on port 6033.
DISK: The persistent SQLite database at /var/lib/proxysql/proxysql.db. SAVE <MODULE> TO DISK copies from MEMORY to DISK. On restart, ProxySQL reads from DISK into MEMORY, then loads to RUNTIME.

The critical asymmetry: LOAD TO RUNTIME and SAVE TO DISK are independent operations. A change can be live (in RUNTIME) but not durable (not in DISK). On restart, DISK wins. Everything in RUNTIME that was not saved is gone.

The config file is a separate bootstrap source. Once proxysql.db exists, the config file is ignored on normal restarts. It is only read when ProxySQL is started with --initial (recreate the database from the config file) or --reload (merge config file settings into the existing database).

flowchart TD
    A["Admin query\nUPDATE / SET"] --> B["MEMORY\nstaging"]
    B -->|"LOAD ... TO RUNTIME"| C["RUNTIME\nlive, port 6033"]
    B -->|"SAVE ... TO DISK"| D["DISK\nproxysql.db"]
    D -->|"On restart: loads here"| B
    B -->|"auto-load on restart"| C
    E["Config file\nproxysql.cnf"] -->|"--initial or --reload"| B

The arrow from MEMORY to DISK via SAVE ... TO DISK is the step operators forget. On restart, data flows from DISK back to MEMORY and then to RUNTIME, silently replacing whatever was live.

Common causes

Cause	What it looks like	First thing to check
Forgot `SAVE ... TO DISK` after incident fix	Change was live for hours or days, then vanished after restart	Compare `mysql_servers` vs `disk.mysql_servers`
Partial save (saved some modules, missed others)	Some changes persisted, others reverted	Check each module individually: servers, users, query rules, variables
Restart with `--reload` or `--initial`	All admin-interface changes lost; config reverted to config file values	Check startup flags in systemd unit or process list
Cluster sync overwrote local change	Change was saved locally but disappeared after peer sync	Check `stats_proxysql_servers_checksums` for divergence
Datadir mismatch	`SAVE` wrote to a different database than the one loaded on restart	Verify datadir path in config and actual `proxysql.db` location

Quick checks

These are read-only queries against the admin interface (default port 6032). None of them modify configuration.

Password on the command line (-padmin) is visible in the process list and shell history. Use a .my.cnf or MYSQL_PWD environment variable in production.

# Confirm whether a restart happened
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name = 'ProxySQL_Uptime';"

# Compare MEMORY (staging) vs RUNTIME (live) for mysql_servers
# Divergence means someone changed config but never ran LOAD TO RUNTIME
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port, max_connections FROM mysql_servers ORDER BY hostgroup_id;" \
  -e "SELECT hostgroup_id, hostname, port, max_connections FROM runtime_mysql_servers ORDER BY hostgroup_id;"

# Compare MEMORY vs DISK for mysql_servers
# Divergence means a restart will revert to what is on DISK
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port FROM mysql_servers ORDER BY hostgroup_id;" \
  -e "SELECT hostgroup_id, hostname, port FROM disk.mysql_servers ORDER BY hostgroup_id;"

# Quick checksum comparison (no row-level diff)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "CHECKSUM TABLE mysql_servers; CHECKSUM TABLE runtime_mysql_servers;"

# In ProxySQL Cluster: check if peers have diverged
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM stats_proxysql_servers_checksums;"

# Check what flags ProxySQL started with
ps aux | grep '[p]roxysql'

How to diagnose it

Confirm the restart happened. Check ProxySQL_Uptime in stats_mysql_global. If it is much lower than expected, a restart occurred. Cross-reference with the systemd journal or your process supervisor.
Identify what reverted. Query runtime_mysql_servers and compare against what you expect. Check runtime_mysql_users for credential reversion. Check runtime_mysql_query_rules for routing rule changes. The disk.* prefix tables show what is persisted: disk.mysql_servers, disk.mysql_users, disk.mysql_query_rules.
Determine which modules were affected. If you changed mysql_servers and mysql_users during the incident but only ran SAVE MYSQL SERVERS TO DISK, the user changes are lost. There is no single command that saves all modules. Each must be saved individually.
Check for config file interference. If someone restarted with --reload or --initial, the config file overwrote the SQLite database. Check the process flags and the ProxySQL error log for messages about config file parsing. Also check whether someone ran LOAD ... FROM CONFIG through the admin interface, which loads config file values into MEMORY and can overwrite staging changes.
Check cluster sync state. If you are running ProxySQL Cluster, a peer may have pushed its configuration to this node. The admin-cluster_mysql_users_save_to_disk variable defaults to true, meaning cluster-synced users are automatically saved to disk after a remote sync and load to runtime. This can surprise operators who expect manual-only persistence. Check stats_proxysql_servers_checksums for divergence.
Verify the datadir. If SAVE ... TO DISK wrote to a different location than where ProxySQL reads on startup, the save was ineffective. The datadir is configured in the ProxySQL configuration file. Verify the path matches the actual proxysql.db location.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`ProxySQL_Uptime`	A restart triggers reload from DISK. Any unsaved RUNTIME changes are lost at this moment.	Sudden drop to a low value
`Servers_table_version`	Increments when the servers table changes via admin query or cluster sync.	Unexpected increment not correlated with a known change
`stats_proxysql_servers_checksums`	In cluster mode, shows whether all nodes have the same config.	Checksum mismatch between peers
`Access_Denied_Wrong_Password`	If credentials reverted on restart, auth failures spike immediately after.	Spike coinciding with uptime reset
Backend status transitions	If monitor credentials reverted, backends go SHUNNED due to failed health checks.	SHUNNED transitions right after restart
`Client_Connections_created`	Empty connection pool after restart causes a connection storm as all clients reconnect.	Spike at the moment uptime resets

Fixes

Re-apply and persist the change

If the change was never saved to disk, re-apply it through the admin interface and save every affected module:

-- Example: re-apply a backend server change
UPDATE mysql_servers SET max_connections=200 WHERE hostgroup_id=1 AND hostname='10.0.0.5';
LOAD MYSQL SERVERS TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

Then verify the save by comparing MEMORY and DISK:

SELECT hostgroup_id, hostname, port, max_connections FROM mysql_servers;
SELECT hostgroup_id, hostname, port, max_connections FROM disk.mysql_servers;

If the rows match, the change is persisted. If they differ, the save failed or the datadir is wrong.

Save all commonly changed modules

There is no SAVE ALL TO DISK command. After an incident, save every module you might have touched:

SAVE MYSQL SERVERS TO DISK;
SAVE MYSQL USERS TO DISK;
SAVE MYSQL QUERY RULES TO DISK;
SAVE MYSQL VARIABLES TO DISK;
SAVE ADMIN VARIABLES TO DISK;

Running all five takes seconds and eliminates the “which module did I forget” problem. If you use the scheduler or REST API features, save those modules too.

Recover from config file overwrite

If someone started ProxySQL with --initial or --reload and the config file was stale, the SQLite database has been overwritten. You need to re-apply all configuration through the admin interface and save to disk.

If the database is corrupted and ProxySQL will not start, deleting proxysql.db and restarting with --initial recreates it from the config file. Any changes that existed only in the database are lost. This is a last resort.

Fix cluster sync issues

If a peer node overwrote your local changes, apply the change on all cluster nodes or on the node that acts as the configuration source. Check admin-cluster_mysql_users_save_to_disk and related admin-cluster_*_save_to_disk variables to understand which modules auto-save after a cluster sync.

Prevention

Post-incident SAVE checklist. Treat SAVE ... TO DISK as part of closing the incident, not as an afterthought. Before declaring an incident resolved, run the five-module save sequence above and verify that MEMORY matches DISK for each module you changed.

Document which modules were changed. During an incident, note every table you modified: mysql_servers, mysql_users, mysql_query_rules, mysql_variables, admin_variables. After the incident, save each one explicitly.

Never edit the config file on a running system. The config file is a bootstrap source. Editing it and restarting will either do nothing (normal restart ignores it) or overwrite everything (--reload or --initial). All runtime configuration changes should go through the admin interface.

Be aware of cluster auto-save behavior. The admin-cluster_mysql_users_save_to_disk variable defaults to true. In a ProxySQL Cluster, synced users are automatically persisted to disk. If you rely on manual-only persistence, this default will surprise you.

Monitor for config divergence. In cluster mode, stats_proxysql_servers_checksums reveals when peers have different configurations. Regularly compare MEMORY and DISK tables for key modules to catch drift before a restart makes it permanent.

How Netdata helps

ProxySQL_Uptime tracking with per-second collection immediately surfaces restarts. Correlating an uptime reset with subsequent auth failures, backend shunning, or connection storms tells you whether the restart triggered a config reversion.
Backend status monitoring catches the downstream effect of reverted monitor credentials. If mysql-monitor_password was lost, backends go SHUNNED within the first monitor check cycle after restart.
Access_Denied_Wrong_Password tracking detects credential reversion. A spike at the exact moment ProxySQL_Uptime resets is a strong signal that mysql_users reverted to stale disk state.
Connection storm detection correlates Client_Connections_created spikes with restart events, helping distinguish a normal cold-start pool fill from a config-induced cascade.
Cluster checksum monitoring surfaces config divergence between ProxySQL peers before a restart makes the split permanent.