$ guides / proxysql / proxysql-cluster-checksum-mismatch ▌

Operations Guides

ProxySQL Cluster checksum mismatch: split-brain routing across proxy peers

ProxySQL Cluster peers exchange configuration via a pull-based sync protocol. When sync stalls or conflicts, peers diverge silently. Clients on proxy A get one set of backends; clients on proxy B get a different set. The same query routes to different hostgroups. Credentials differ. Query rules match differently.

The detection signal is stats_proxysql_servers_checksums. Each peer computes a checksum per config module. When peers agree, checksums match. When they disagree, diff_check climbs for the affected module. Persistent mismatch means the cluster is not converging.

This applies only to ProxySQL Cluster deployments. Standalone instances leave stats_proxysql_servers_checksums unpopulated.

What this means

At least two peers have different runtime configuration for the same module. The operational impact depends on which module diverged:

mysql_servers divergence: clients routed to different backend sets. Write queries may land on different primaries. Traffic distribution assumptions break.
mysql_query_rules divergence: the same query matches different rules on different proxies. Read/write splitting and cache rules break. Traffic lands on wrong hostgroups.
mysql_users divergence: different credentials in effect. Some proxies reject connections that others accept.
admin_variables or mysql_variables divergence: runtime behavior (monitor intervals, shun thresholds, multiplexing) differs across peers, producing inconsistent backend health decisions.

Six modules generate checksums in current ProxySQL releases: mysql_query_rules, mysql_servers, mysql_users, proxysql_servers, admin_variables, and mysql_variables. ProxySQL v2.1.0 extended cluster synchronization to Galera, Group Replication, and Aurora, so the six-module list applies to current releases rather than all historical 2.0.x releases.

Each node polls peers at admin-cluster_check_interval_ms (default 1000 ms). When a mismatch persists for admin-cluster_*_diffs_before_sync consecutive checks (default 3), the node pulls config from the peer with the highest epoch, provided that peer’s version is greater than 1. A node at version=1 is never a sync source. This prevents a freshly restarted node with stale disk config from overwriting the cluster.

Convergence should happen within roughly admin-cluster_check_interval_ms multiplied by 3. When checksums stay mismatched beyond that window, the pull is failing or no valid source exists.

flowchart TD
    A["Checksum mismatch detected"] --> B{"All peers at
version 1?"}
    B -->|Yes| C["No valid sync source.
Rolling restart trap."]
    B -->|No| D{"diff_check climbing
without convergence?"}
    D -->|Yes| E{"Which module
diverges?"}
    D -->|No, stable diff| F["Stable divergence.
Network or config conflict."]
    E -->|"mysql_servers"| G["Check auxiliary
hostgroup tables"]
    E -->|"mysql_query_rules"| H["Check rule order
and checksum variables"]
    E -->|"mysql_users"| I["Check credential
rotation window"]
    G --> J["Load authoritative config
on one node"]
    H --> J
    I --> J
    C --> J
    F --> J

Common causes

Cause	What it looks like	First thing to check
Simultaneous divergent config changes	Two or more peers have different checksums for the same module. Both have high epoch values. `diff_check` climbs but never resolves.	Check who made admin changes on which proxy and when.
version=1 on all peers after rolling restart	Every peer shows `version=1` for one or more modules. No peer qualifies as sync source.	`SELECT hostname, name, version FROM stats_proxysql_servers_checksums;`
mysql_servers auxiliary table sync gap	`diff_check` for `mysql_servers` climbs indefinitely. Checksums never converge. Galera or Group Replication hostgroup tables differ between peers.	Compare `mysql_galera_hostgroups` or `mysql_group_replication_hostgroups` across peers.
Network partition between peers	Peers cannot reach each other on the admin port (6032). `stats_proxysql_servers_metrics` shows connection failures.	Check peer connectivity in `proxysql_servers` table and network paths.
Checksum variables disabled on some peers	Some peers report a zero checksum for a module. Others report real checksums.	`SELECT variable_name, variable_value FROM global_variables WHERE variable_name LIKE 'admin-checksum_%';`

Quick checks

Run these against each peer in the cluster, not just one. The examples use default admin credentials (admin/admin); adjust for your environment.

# 1. View all checksums across all peers for all modules
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostname, port, name, version, epoch, checksum, diff_check FROM stats_proxysql_servers_checksums ORDER BY name, hostname;"

# 2. Find modules where peers disagree (count distinct checksums per module)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT name, COUNT(DISTINCT checksum) AS distinct_checksums, MAX(diff_check) AS max_diff_check FROM stats_proxysql_servers_checksums GROUP BY name;"

# 3. Check which peers are at version=1 (no sync source available)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostname, port, name, version FROM stats_proxysql_servers_checksums WHERE version = 1;"

# 4. Check cluster check interval and diffs-before-sync settings
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM global_variables WHERE variable_name LIKE 'admin-cluster_%' ORDER BY variable_name;"

# 5. Verify checksum generation is enabled for each module
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM global_variables WHERE variable_name LIKE 'admin-checksum_%';"

# 6. Check peer connectivity and sync metrics
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM stats_proxysql_servers_metrics;"

# 7. List configured cluster peers
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM proxysql_servers;"

How to diagnose it

Identify which modules diverge. Run check 2 above. Any module with distinct_checksums > 1 is divergent. Focus first on mysql_servers and mysql_query_rules because these directly affect routing.
Check version values. If all peers show version=1 for a divergent module, no peer qualifies as sync source. This is the rolling restart trap.
Check diff_check trends. Steady increment without convergence means sync is attempted but failing. A stable diff_check means the node stopped trying, likely because no valid source exists or it gave up after repeated failures.
Determine which peer has the authoritative config. Compare runtime_mysql_servers, runtime_mysql_query_rules, and runtime_mysql_users across peers. The peer with the correct config should be the sync source. Verify its version is greater than 1.
Check for the auxiliary table sync gap. If mysql_servers checksums never converge, compare mysql_galera_hostgroups, mysql_group_replication_hostgroups, and mysql_replication_hostgroups across peers. The mysql_servers checksum incorporates these auxiliary tables, but sync may not pull all of them.
Verify peer connectivity. Check stats_proxysql_servers_metrics for connection errors. Verify network paths on the admin port (default 6032).

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Distinct checksums per module in `stats_proxysql_servers_checksums`	Direct measure of config divergence across peers	More than 1 distinct checksum for `mysql_servers` or `mysql_query_rules`
`diff_check` counter per module per peer	Shows whether sync is being attempted and failing	Steady growth without convergence beyond the convergence window
`version` per module per peer	Identifies whether a valid sync source exists	All peers at `version=1` means no source available
`epoch` per module per peer	Shows which peer last changed config; highest epoch is preferred sync source	Two peers with similar high epochs indicates competing changes
`stats_proxysql_servers_metrics`	Peer connectivity and sync operation results	Connection failures or sync errors between peers
Client-reported inconsistent behavior	Application teams report queries succeeding on one proxy but failing on another	Correlate reports with specific ProxySQL instances behind the load balancer

Fixes

Fix: simultaneous divergent configs

When different configurations are loaded on multiple proxies, the cluster cannot resolve the conflict automatically. Designate one peer as the source of truth.

On the peer with the correct config, load the divergent module to runtime to bump its epoch:

# Replace MYSQL SERVERS with the module that diverged (MYSQL QUERY RULES, MYSQL USERS, etc.)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "LOAD MYSQL SERVERS TO RUNTIME;"

This increments version and epoch on that peer, making it the preferred sync source. Other peers should pull within admin-cluster_check_interval_ms multiplied by 3. After convergence, persist on all peers:

mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SAVE MYSQL SERVERS TO DISK;"

Only load and save the modules that diverged. Loading unrelated modules unnecessarily bumps their checksums and can mask the real divergence.

Fix: version=1 after rolling restart

When all peers restart in sequence and load config from disk, each resets to version=1. The version guard prevents any from acting as sync source. Pick one peer and load the divergent module to runtime:

mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "LOAD MYSQL SERVERS TO RUNTIME;"

This sets version above 1, making the peer a valid sync source.

If the config on disk was stale because changes were loaded to runtime but never saved before restart, the authoritative config is lost. Reconstruct it manually on one peer, load to runtime, then let the cluster sync.

Fix: auxiliary table sync gap

If you use mysql_galera_hostgroups, mysql_group_replication_hostgroups, or mysql_replication_hostgroups, and the mysql_servers checksum never converges, sync may not be pulling all auxiliary tables.

Current ProxySQL Cluster code pairs the mysql_servers sync query with the related replication, Group Replication, Galera, Aurora, hostgroup-attribute, SSL-parameter, and AWS RDS blue/green tables. On each divergent peer:

mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;"

If divergence recurs immediately after sync, the auxiliary table contents genuinely differ between peers. Fix the config on the authoritative peer first, then let sync propagate.

Fix: network partition

If peers cannot reach each other, no sync is possible. Check:

Admin port (default 6032) is reachable between proxy hosts
Firewall rules allow inter-proxy traffic on the admin port
Hostnames or IP addresses in proxysql_servers are correct and resolve from each peer

After restoring connectivity, convergence should happen within admin-cluster_check_interval_ms multiplied by 3.

Prevention

Make config changes on one peer only. Designate a primary admin peer. Load to runtime and let cluster sync propagate. Never make simultaneous changes on multiple peers.
Always SAVE TO DISK after loading to runtime. A restart resets version to 1 and loads from disk. If disk config is stale, the node comes back with wrong config and version=1.
Monitor stats_proxysql_servers_checksums continuously. Alert on any module with more than 1 distinct checksum sustained beyond the convergence window.
After rolling restarts, verify convergence. Check that at least one peer has version greater than 1 for each module. If all peers are at version=1, manually load one module on the designated peer.
Audit auxiliary hostgroup tables during planned maintenance. If you use Galera or Group Replication hostgroups, verify these tables are identical across peers.
Avoid concurrent admin queries against stats_proxysql_servers_checksums on ProxySQL 2.5.5. Issue #4386 records crashes from concurrent access and the UNIQUE constraint; maintainer-linked issue #4415 identifies the affected v2.5.2-v2.5.5 deadlock and was fixed after that range.

How Netdata helps

Netdata collects ProxySQL metrics per-second, making divergence visible immediately rather than at a slower polling interval. The diff_check trend is the key signal: steady growth without convergence means the cluster cannot self-heal.
Correlate checksum divergence with ProxySQL restart events (via ProxySQL_Uptime drops) to distinguish rolling restart traps from genuine config conflicts. If version=1 appears on all peers after uptime resets, the cause is restart-related.
When mysql_servers checksums diverge, Netdata’s per-backend metrics (connection pool usage, backend status, ConnERR rates) let you confirm which proxy has the correct backend set by comparing operational behavior, not just config tables.
Netdata’s ML anomaly detection can flag unusual diff_check patterns before a human-defined threshold fires, giving early warning that cluster sync is degrading.
For split-brain routing incidents, correlating checksum mismatch timing with client connection patterns (Client_Connections_connected, Client_Connections_aborted) across proxy instances confirms that clients are experiencing the routing divergence.