ProxySQL backend SHUNNED: why a healthy backend gets pulled out of rotation

You see status=SHUNNED on a backend in stats_mysql_connection_pool or runtime_mysql_servers. The backend itself looks healthy when you connect directly. Queries may be failing or routing to fewer backends than expected, but the MySQL server is up and accepting connections.

SHUNNED is ProxySQL’s protection mechanism, not a bug. When a backend generates mysql-shun_on_failures connection errors within one second, or when its replication lag exceeds the configured max_replication_lag, ProxySQL temporarily removes it from rotation. The backend typically returns to ONLINE on its own within mysql-shun_recovery_time_sec (default 10 seconds), provided there is activity in the connection pool for that hostgroup.

The operational question is not “why is it SHUNNED?” but “is this transient protection, or is something repeatedly triggering it?” A single SHUNNED backend is a TICKET. Zero ONLINE backends in a hostgroup is a PAGE.

What this means

ProxySQL’s monitor module continuously health-checks backends via connect, ping, read-only, and replication lag probes. The results drive status transitions in the hostgroup manager. When the monitor detects that a backend has accumulated too many failures or is too far behind on replication, it flips that backend’s status to SHUNNED.

The two triggers are distinct:

1. Connection error threshold: If mysql-shun_on_failures (default 5) errors occur within one second, the backend is shunned. These errors come from ConnERR in stats_mysql_connection_pool: failed connect() calls, authentication failures, backend max_connections exhaustion, or network unreachability.

2. Replication lag threshold: If max_replication_lag is set to a non-zero value in mysql_servers for a backend (default is 0, meaning disabled), and the monitor detects Seconds_Behind_Master exceeding that value, the backend is shunned until replication catches up.

Recovery is not purely timer-based. The backend is brought back ONLINE only after mysql-shun_recovery_time_sec has elapsed AND there is activity in the connection pool for the specific hostgroup. A hostgroup with no client traffic will not recover shunned nodes automatically.

flowchart TD
    A["Backend ONLINE"] -->|"ConnERR >= shun_on_failures\nwithin 1 second"| B["Backend SHUNNED"]
    A -->|"replication lag >\nmax_replication_lag"| B
    B -->|"recovery_time elapsed\nAND pool activity exists"| A
    B -->|"no pool activity\nfor hostgroup"| C["Stays SHUNNED"]
    C -->|"client traffic resumes"| A

SHUNNED prevents new connections but does NOT kill existing ones. Queries already in flight on a shunned backend continue to execute. Only the SHUNNED_REPLICATION_LAG state for Group Replication, which ProxySQL uses from v2.3.0 onward in its two-stage flow, terminates existing connections. If you see ConnUsed > 0 on a SHUNNED backend, those are queries that were routed before the status change.

Common causes

Quick checks

Run these against the admin interface (default port 6032). All are read-only.

# Check current backend statuses in the connection pool
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree, ConnOK, ConnERR FROM stats_mysql_connection_pool;"

# Check runtime server configuration including max_replication_lag
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port, status, max_connections, max_replication_lag FROM runtime_mysql_servers;"

# Check monitor ping and connect results for recent failures
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_ping_log ORDER BY time_start_us DESC LIMIT 50;"
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_connect_log ORDER BY time_start_us DESC LIMIT 50;"

# Check replication lag readings from the monitor
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_replication_lag_log ORDER BY time_start_us DESC LIMIT 20;"

# Check monitor check counters (OK vs ERR rates)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name LIKE 'MySQL_Monitor_%check%';"

# Check current shun-related variables
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM global_variables WHERE variable_name IN ('mysql-shun_on_failures','mysql-shun_recovery_time_sec');"

# Check queries affected by mid-flight status changes
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('backend_lagging_during_query','backend_offline_during_query');"

How to diagnose it

1. Identify which backends are SHUNNED and in which hostgroups. A backend can be SHUNNED in one hostgroup but ONLINE in another if the max_replication_lag threshold differs per entry in mysql_servers. Query stats_mysql_connection_pool for the full picture, not just runtime_mysql_servers.

2. Check whether ConnERR is the trigger. Look at ConnERR in stats_mysql_connection_pool for the affected backend. If it is rising, the shunning is error-driven. Cross-reference with monitor.mysql_server_connect_log and monitor.mysql_server_ping_log to find the specific error type: connection refused, timeout, auth failure.

3. Check whether replication lag is the trigger. If only reader hostgroup members are SHUNNED and the writer is fine, look at monitor.mysql_server_replication_lag_log. Verify that max_replication_lag is set to a non-zero value for those backends in runtime_mysql_servers. If max_replication_lag = 0, lag checking is disabled and this is not the cause.

4. Determine if the backend is actually unhealthy. Connect directly to the backend MySQL, bypassing ProxySQL. Run SHOW SLAVE STATUS, check SHOW PROCESSLIST, and verify the backend is accepting connections. If the backend is healthy when accessed directly but SHUNNED in ProxySQL, the issue is in the monitor path: network between proxy and backend, monitor credentials, or aggressive thresholds.

5. Check for flapping. If the backend cycles between ONLINE and SHUNNED rapidly (more than 3 transitions in 10 minutes), look at the monitor log timestamps. Flapping indicates the monitor intervals or shun_on_failures threshold are too aggressive for the network conditions, or the backend is intermittently unresponsive to health checks while serving queries normally.

6. Verify pool activity exists for recovery. If a backend is stuck SHUNNED longer than mysql-shun_recovery_time_sec, check whether there is actual client traffic to that hostgroup. ConnUsed == 0 and ConnFree == 0 in stats_mysql_connection_pool for that hostgroup means no traffic is reaching it, so recovery will not trigger.

7. Check self-tuning for single-backend hostgroups. mysql-shun_recovery_time_sec self-tunes to 1 second when only one backend exists in a hostgroup and the configured value is greater than 1. If this does not work in your version, a single-writer hostgroup can experience repeated 10-second outages.

Metrics and signals to monitor

Fixes

When connection errors are the trigger

If ConnERR is rising, the backend is rejecting connections from ProxySQL. Common causes:

• Backend max_connections exhausted: ProxySQL competes with direct connections, replication threads, and other proxy instances for backend MySQL connections. Check SHOW GLOBAL STATUS LIKE 'Threads_connected' on the backend against its max_connections. Adjust ProxySQL’s per-backend max_connections in mysql_servers to leave headroom.
• Monitor credentials expired: If mysql-monitor_username or mysql-monitor_password are invalid on the backend, monitor checks fail and ProxySQL shuns the backend. Update credentials, then run LOAD MYSQL VARIABLES TO RUNTIME; SAVE MYSQL VARIABLES TO DISK;.
• Network partition: Check connectivity from the ProxySQL host to the backend with mtr or a direct mysql client connection. Cloud networking can introduce intermittent packet loss or latency spikes that cross the shun_on_failures threshold.
• MySQL wait_timeout closing pooled connections: If the backend closes idle connections after wait_timeout and ProxySQL tries to reuse them, the failed reuse counts as a ConnERR. Set mysql-connection_max_age_ms in ProxySQL to a value shorter than the backend’s wait_timeout (converted to milliseconds). The default of 0 means no age-based recycling.

When replication lag is the trigger

If replicas are being shunned due to lag:

• Verify max_replication_lag is set appropriately in mysql_servers. The default of 0 disables lag checking. A value too low for your workload causes constant shunning during normal replication delay spikes.
• Check SHOW SLAVE STATUS on the affected replica. If Seconds_Behind_Master is genuinely high, the replica cannot keep up with the write workload. This is a backend capacity issue, not a ProxySQL issue.
• Note that Seconds_Behind_Master is unreliable during large transactions (shows 0, then jumps). If you see replicas shunned immediately after a large batch write, the lag may have been masked during the transaction.
• For Group Replication, ProxySQL compares the backend’s max_transactions_behind table value with mysql-monitor_groupreplication_max_transactions_behind_count, constrained by mysql-monitor_groupreplication_max_transactions_behind_for_read_only. Since v2.3.0, a server first becomes SHUNNED at the configured count and becomes SHUNNED_REPLICATION_LAG—terminating its connections—at twice that count.

When the backend flaps between ONLINE and SHUNNED

Flapping means the monitor is too sensitive relative to actual backend stability:

• Increase mysql-monitor_ping_interval (default 8000ms) to reduce check frequency. Aggressive intervals (e.g., 500ms) on a network with jitter will produce enough transient failures to cross shun_on_failures.
• Increase mysql-shun_on_failures from the default of 5. This variable cannot be disabled by setting it to a special value. To effectively disable shunning, set it to a very large number (e.g., 10000000), but this removes the protection entirely.
• Check whether the backend is under resource pressure during health checks. A backup, large analytical query, or ANALYZE TABLE can make the backend briefly unresponsive to mysql_ping() while still serving other queries.

When a single-backend hostgroup gets stuck

A hostgroup with only one backend is the most dangerous shunning scenario because there is nowhere to redirect traffic:

• mysql-shun_recovery_time_sec should self-tune to 1 second for single-backend hostgroups. If this does not work in your version, a single writer can be shunned for the full 10 seconds repeatedly, causing cascading “Too many connections” errors.
• Recovery requires pool activity. If the hostgroup has no traffic (e.g., a standby writer hostgroup), the shunned backend will not recover automatically. Consider routing a lightweight health-check query through the hostgroup periodically.
• For Galera setups using mysql_galera_hostgroups with max_writers: extra writer-hostgroup entries are intentionally shown as SHUNNED while preserving their existing connections. shunned_automatic is an internal state, not a runtime_mysql_servers column. If the active writer fails, verify placement and promote a healthy backup manually; a 2021 issue report #3618 describes skipped automatic promotion on v2.2.x and remains open.

When you need to force a backend back online

runtime_mysql_servers reflects RUNTIME configuration; update mysql_servers and reload instead of editing it directly.

Toggle the backend status through mysql_servers to force a runtime reload that resets the monitor state:

UPDATE mysql_servers SET status='OFFLINE_SOFT'
  WHERE hostgroup_id=<hg> AND hostname='<host>' AND port=<port>;
LOAD MYSQL SERVERS TO RUNTIME;
UPDATE mysql_servers SET status='ONLINE'
  WHERE hostgroup_id=<hg> AND hostname='<host>' AND port=<port>;
LOAD MYSQL SERVERS TO RUNTIME;

WARNING: This is a temporary fix. If the underlying condition persists, the monitor will re-shun the backend on the next check cycle. Always identify and address the root cause.

Prevention

• Tune mysql-shun_on_failures based on your network characteristics. The default of 5 failures per second is reasonable for low-latency LANs but too sensitive for cross-AZ or cross-region setups with occasional jitter.
• Set max_replication_lag deliberately. Leaving it at 0 disables lag-based shunning entirely. Setting it too low causes unnecessary shunning. Base the value on your application’s staleness tolerance.
• Track monitor check error rates. Monitor MySQL_Monitor_connect_check_ERR and MySQL_Monitor_ping_check_ERR. Rising error rates precede SHUNNED transitions and give you early warning.
• Verify recovery works under low traffic. In staging environments with sparse traffic, confirm that shunned backends actually recover. If pool activity is too low, recovery never triggers.
• Keep monitor credentials in sync. Credential rotation that updates mysql_users but not mysql-monitor_password will cause monitor check failures and spurious shunning.
• Review replication lag check timeout. Use mysql-monitor_replication_lag_timeout (default 1000ms) and mysql-monitor_replication_lag_count (default 1). A too-short timeout can prevent ProxySQL from measuring lag reliably.

How Netdata helps

Netdata’s ProxySQL collector provides per-second visibility into the signals that drive SHUNNED transitions:

• Backend status from stats_mysql_connection_pool: See exactly when a backend transitions to SHUNNED and how long it stays there, without polling gaps that miss transient events.
• ConnERR rates per backend: Correlate connection error spikes with SHUNNED transitions to confirm the trigger.
• Monitor check counters: Track MySQL_Monitor_*_check_OK and _ERR rates to detect rising failure rates before they trigger shunning.
• backend_lagging_during_query and backend_offline_during_query: Quantify queries affected by status changes mid-flight, measuring user impact beyond the status flag.
• ML anomaly detection: Baselines normal behavior per metric to distinguish transient SHUNNED events from sustained degradation.

Per-second granularity matters because shun and recovery events can complete within a 10-second window, making slower polling intervals miss the full cycle.

Related guides

• ProxySQL zero ONLINE backends in a hostgroup: total outage for that traffic class
• ProxySQL OFFLINE_SOFT vs OFFLINE_HARD vs SHUNNED: what each backend status means
• ProxySQL backend flapping between ONLINE and SHUNNED: monitor-induced oscillation
• ProxySQL monitor check failures: connect, ping, read-only, and replication-lag probes failing
• ProxySQL MySQL_Monitor_Workers is zero: health checks stopped and status is stale
• ProxySQL monitoring checklist: the signals every production proxy needs
• How ProxySQL actually works in production: a mental model for operators
• ProxySQL monitoring maturity model: from survival to expert
• ProxySQL backend connection pool exhausted: queries queuing for a free connection
• ProxySQL error 1040 Too many connections: the backend MySQL rejecting the pool
• ProxySQL error 9001 Max connect timeout reached while reaching hostgroup
• ProxySQL ConnPool_get_conn_failure rising: the most direct pool-starvation signal