You see status=SHUNNED on a backend in stats_mysql_connection_pool or runtime_mysql_servers. The backend itself looks healthy when you connect directly. Queries may be failing or routing to fewer backends than expected, but the MySQL server is up and accepting connections.

SHUNNED is ProxySQL’s protection mechanism, not a bug. When a backend generates more than mysql-shun_on_failures connection errors within one second, or when its replication lag exceeds the configured max_replication_lag, ProxySQL temporarily removes it from rotation. The backend typically returns to ONLINE on its own within mysql-shun_recovery_time_sec (default 10 seconds), provided there is activity in the connection pool for that hostgroup.

The operational question is not “why is it SHUNNED?” but “is this transient protection, or is something repeatedly triggering it?” A single SHUNNED backend is a TICKET. Zero ONLINE backends in a hostgroup is a PAGE.

What this means

ProxySQL’s monitor module continuously health-checks backends via connect, ping, read-only, and replication lag probes. The results drive status transitions in the hostgroup manager. When the monitor detects that a backend has accumulated too many failures or is too far behind on replication, it flips that backend’s status to SHUNNED.

The two triggers are distinct:

  1. Connection error threshold: If more than mysql-shun_on_failures (default 5) errors occur within one second, the backend is shunned. These errors come from ConnERR in stats_mysql_connection_pool: failed connect() calls, authentication failures, backend max_connections exhaustion, or network unreachability.

  2. Replication lag threshold: If max_replication_lag is set to a non-zero value in mysql_servers for a backend (default is 0, meaning disabled), and the monitor detects Seconds_Behind_Master exceeding that value, the backend is shunned until replication catches up.

Recovery is not purely timer-based. The backend is brought back ONLINE only after mysql-shun_recovery_time_sec has elapsed AND there is activity in the connection pool for the specific hostgroup. A hostgroup with no client traffic will not recover shunned nodes automatically.

flowchart TD
    A["Backend ONLINE"] -->|"ConnERR > shun_on_failures\nwithin 1 second"| B["Backend SHUNNED"]
    A -->|"replication lag >\nmax_replication_lag"| B
    B -->|"recovery_time elapsed\nAND pool activity exists"| A
    B -->|"no pool activity\nfor hostgroup"| C["Stays SHUNNED"]
    C -->|"client traffic resumes"| A

SHUNNED prevents new connections but does NOT kill existing ones. Queries already in flight on a shunned backend continue to execute. Only the SHUNNED_REPLICATION_LAG state (Group Replication, v2.3.0+) terminates existing connections. If you see ConnUsed > 0 on a SHUNNED backend, those are queries that were routed before the status change.

Common causes

CauseWhat it looks likeFirst thing to check
Connection errors exceeding thresholdConnERR rising rapidly, then status flips to SHUNNEDmonitor.mysql_server_connect_log and monitor.mysql_server_ping_log
Replication lag exceeding max_replication_lagOnly replicas shunned, writer stays ONLINESHOW SLAVE STATUS on the affected backend
Monitor-induced flappingRapid ONLINE to SHUNNED cycling, backend healthy when checked directlyMonitor ping log for alternating success/failure pattern
MySQL wait_timeout closing pooled connectionsPeriodic ConnERR spikes matching the wait_timeout intervalBackend wait_timeout vs ProxySQL mysql-connection_max_age_ms
Stale backend entries after orchestration churnConnERR on backends that no longer existruntime_mysql_servers for entries pointing to removed hosts
Galera max_writers shunning (Issue #3618)Nodes SHUNNED due to max_writers, never auto-recovershunned_automatic flag in runtime_mysql_servers

Quick checks

Run these against the admin interface (default port 6032). All are read-only.

# Check current backend statuses in the connection pool
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree, ConnOK, ConnERR FROM stats_mysql_connection_pool;"

# Check runtime server configuration including max_replication_lag
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port, status, max_connections, max_replication_lag FROM runtime_mysql_servers;"

# Check monitor ping and connect results for recent failures
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_ping_log ORDER BY time_start_us DESC LIMIT 50;"
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_connect_log ORDER BY time_start_us DESC LIMIT 50;"

# Check replication lag readings from the monitor
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_replication_lag_log ORDER BY time_start_us DESC LIMIT 20;"

# Check monitor check counters (OK vs ERR rates)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name LIKE 'MySQL_Monitor_%check%';"

# Check current shun-related variables
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM global_variables WHERE variable_name IN ('mysql-shun_on_failures','mysql-shun_recovery_time_sec');"

# Check queries affected by mid-flight status changes
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('backend_lagging_during_query','backend_offline_during_query');"

How to diagnose it

  1. Identify which backends are SHUNNED and in which hostgroups. A backend can be SHUNNED in one hostgroup but ONLINE in another if the max_replication_lag threshold differs per entry in mysql_servers. Query stats_mysql_connection_pool for the full picture, not just runtime_mysql_servers.

  2. Check whether ConnERR is the trigger. Look at ConnERR in stats_mysql_connection_pool for the affected backend. If it is rising, the shunning is error-driven. Cross-reference with monitor.mysql_server_connect_log and monitor.mysql_server_ping_log to find the specific error type: connection refused, timeout, auth failure.

  3. Check whether replication lag is the trigger. If only reader hostgroup members are SHUNNED and the writer is fine, look at monitor.mysql_server_replication_lag_log. Verify that max_replication_lag is set to a non-zero value for those backends in runtime_mysql_servers. If max_replication_lag = 0, lag checking is disabled and this is not the cause.

  4. Determine if the backend is actually unhealthy. Connect directly to the backend MySQL, bypassing ProxySQL. Run SHOW SLAVE STATUS, check SHOW PROCESSLIST, and verify the backend is accepting connections. If the backend is healthy when accessed directly but SHUNNED in ProxySQL, the issue is in the monitor path: network between proxy and backend, monitor credentials, or aggressive thresholds.

  5. Check for flapping. If the backend cycles between ONLINE and SHUNNED rapidly (more than 3 transitions in 10 minutes), look at the monitor log timestamps. Flapping indicates the monitor intervals or shun_on_failures threshold are too aggressive for the network conditions, or the backend is intermittently unresponsive to health checks while serving queries normally.

  6. Verify pool activity exists for recovery. If a backend is stuck SHUNNED longer than mysql-shun_recovery_time_sec, check whether there is actual client traffic to that hostgroup. ConnUsed == 0 and ConnFree == 0 in stats_mysql_connection_pool for that hostgroup means no traffic is reaching it, so recovery will not trigger.

  7. Check self-tuning for single-backend hostgroups. mysql-shun_recovery_time_sec self-tunes to 1 second when only one backend exists in a hostgroup and the configured value is greater than 1. If this does not work in your version, a single-writer hostgroup can experience repeated 10-second outages.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
status in stats_mysql_connection_poolThe actual SHUNNED state per backendAny backend not ONLINE; zero ONLINE in a hostgroup
ConnERR per backendCumulative failed connection attempts; drives error-based shunningRising rate preceding a SHUNNED transition
MySQL_Monitor_ping_check_ERRMonitor detecting backend as unresponsiveSustained increase or spikes correlating with SHUNNED events
MySQL_Monitor_connect_check_ERRMonitor cannot establish connections to backendAny sustained failures
MySQL_Monitor_replication_lag_check_ERRReplication lag check failing or timing outCorrelates with lag-based shunning
backend_lagging_during_queryQueries that hit a backend detected as lagging after routingNon-zero values indicate stale reads may have occurred
backend_offline_during_queryQueries that failed because backend went offline mid-executionNon-zero values mean queries were killed by status changes
ConnPool_get_conn_failurePool could not provide a backend connectionIndicates pool starvation beyond a single SHUNNED backend
Questions rateOverall query throughputSudden drop may indicate capacity loss from shunning

Fixes

When connection errors are the trigger

If ConnERR is rising, the backend is rejecting connections from ProxySQL. Common causes:

  • Backend max_connections exhausted: ProxySQL competes with direct connections, replication threads, and other proxy instances for backend MySQL connections. Check SHOW GLOBAL STATUS LIKE 'Threads_connected' on the backend against its max_connections. Adjust ProxySQL’s per-backend max_connections in mysql_servers to leave headroom.
  • Monitor credentials expired: If mysql-monitor_username or mysql-monitor_password are invalid on the backend, monitor checks fail and ProxySQL shuns the backend. Update credentials, then run LOAD MYSQL VARIABLES TO RUNTIME; SAVE MYSQL VARIABLES TO DISK;.
  • Network partition: Check connectivity from the ProxySQL host to the backend with mtr or a direct mysql client connection. Cloud networking can introduce intermittent packet loss or latency spikes that cross the shun_on_failures threshold.
  • MySQL wait_timeout closing pooled connections: If the backend closes idle connections after wait_timeout and ProxySQL tries to reuse them, the failed reuse counts as a ConnERR. Set mysql-connection_max_age_ms in ProxySQL to a value shorter than the backend’s wait_timeout (converted to milliseconds). The default of 0 means no age-based recycling.

When replication lag is the trigger

If replicas are being shunned due to lag:

  • Verify max_replication_lag is set appropriately in mysql_servers. The default of 0 disables lag checking. A value too low for your workload causes constant shunning during normal replication delay spikes.
  • Check SHOW SLAVE STATUS on the affected replica. If Seconds_Behind_Master is genuinely high, the replica cannot keep up with the write workload. This is a backend capacity issue, not a ProxySQL issue.
  • Note that Seconds_Behind_Master is unreliable during large transactions (shows 0, then jumps). If you see replicas shunned immediately after a large batch write, the lag may have been masked during the transaction.
  • For Group Replication, shunning uses max_transactions_behind. A two-stage process applies: first SHUNNED (graceful, connections preserved), then SHUNNED_REPLICATION_LAG (connections terminated) if lag persists.

When the backend flaps between ONLINE and SHUNNED

Flapping means the monitor is too sensitive relative to actual backend stability:

  • Increase mysql-monitor_ping_interval (default 1000ms) to reduce check frequency. Aggressive intervals (e.g., 500ms) on a network with jitter will produce enough transient failures to cross shun_on_failures.
  • Increase mysql-shun_on_failures from the default of 5. This variable cannot be disabled by setting it to a special value. To effectively disable shunning, set it to a very large number (e.g., 10000000), but this removes the protection entirely.
  • Check whether the backend is under resource pressure during health checks. A backup, large analytical query, or ANALYZE TABLE can make the backend briefly unresponsive to mysql_ping() while still serving other queries.

When a single-backend hostgroup gets stuck

A hostgroup with only one backend is the most dangerous shunning scenario because there is nowhere to redirect traffic:

  • mysql-shun_recovery_time_sec should self-tune to 1 second for single-backend hostgroups. If this does not work in your version, a single writer can be shunned for the full 10 seconds repeatedly, causing cascading “Too many connections” errors.
  • Recovery requires pool activity. If the hostgroup has no traffic (e.g., a standby writer hostgroup), the shunned backend will not recover automatically. Consider routing a lightweight health-check query through the hostgroup periodically.
  • For Galera setups using mysql_galera_hostgroups with max_writers: nodes SHUNNED due to max_writers may not set the shunned_automatic flag. When the primary writer fails, ProxySQL only attempts recovery on nodes with shunned_automatic=true. Backup writers remain SHUNNED indefinitely and all writes are blocked.

When you need to force a backend back online

Toggle the backend status through mysql_servers to force a runtime reload that resets the monitor state:

UPDATE mysql_servers SET status='OFFLINE_SOFT'
  WHERE hostgroup_id=<hg> AND hostname='<host>' AND port=<port>;
LOAD MYSQL SERVERS TO RUNTIME;
UPDATE mysql_servers SET status='ONLINE'
  WHERE hostgroup_id=<hg> AND hostname='<host>' AND port=<port>;
LOAD MYSQL SERVERS TO RUNTIME;

WARNING: This is a temporary fix. If the underlying condition persists, the monitor will re-shun the backend on the next check cycle. Always identify and address the root cause.

Prevention

  • Tune mysql-shun_on_failures based on your network characteristics. The default of 5 failures per second is reasonable for low-latency LANs but too sensitive for cross-AZ or cross-region setups with occasional jitter.
  • Set max_replication_lag deliberately. Leaving it at 0 disables lag-based shunning entirely. Setting it too low causes unnecessary shunning. Base the value on your application’s staleness tolerance.
  • Track monitor check error rates. Monitor MySQL_Monitor_connect_check_ERR and MySQL_Monitor_ping_check_ERR. Rising error rates precede SHUNNED transitions and give you early warning.
  • Verify recovery works under low traffic. In staging environments with sparse traffic, confirm that shunned backends actually recover. If pool activity is too low, recovery never triggers.
  • Keep monitor credentials in sync. Credential rotation that updates mysql_users but not mysql-monitor_password will cause monitor check failures and spurious shunning.
  • Review replication lag check timeout. If the lag check times out before completing, ProxySQL may fail to detect actual lag or produce inconsistent shunning decisions.

How Netdata helps

Netdata’s ProxySQL collector provides per-second visibility into the signals that drive SHUNNED transitions:

  • Backend status from stats_mysql_connection_pool: See exactly when a backend transitions to SHUNNED and how long it stays there, without polling gaps that miss transient events.
  • ConnERR rates per backend: Correlate connection error spikes with SHUNNED transitions to confirm the trigger.
  • Monitor check counters: Track MySQL_Monitor_*_check_OK and _ERR rates to detect rising failure rates before they trigger shunning.
  • backend_lagging_during_query and backend_offline_during_query: Quantify queries affected by status changes mid-flight, measuring user impact beyond the status flag.
  • ML anomaly detection: Baselines normal behavior per metric to distinguish transient SHUNNED events from sustained degradation.

Per-second granularity matters because shun and recovery events can complete within a 10-second window, making slower polling intervals miss the full cycle.