ProxySQL zero ONLINE backends in a hostgroup: total outage for that traffic class

When ProxySQL reports zero ONLINE backends in a hostgroup, every query routed to that hostgroup fails immediately. If the affected hostgroup handles writes, all INSERT, UPDATE, and DELETE operations fail. If it is a reader hostgroup, read queries either error out or, depending on query rules, flood a writer hostgroup not sized for that load.

ProxySQL logs Hostgroup <id> has no servers available!; clients may instead see ProxySQL error 9001, “Max connect timeout reached while reaching hostgroup…”, or a generic connection timeout. Application error rates spike. Questions drops toward zero while Client_Connections_aborted rises.

This is a PAGE-level condition, but only under gating criteria that exclude false positives. The diagnostic path: confirm the outage is real (not cold start or idle), determine whether backends are reachable, then inspect monitor check results to identify the specific failure mode.

What this means

ProxySQL tracks four backend states per hostgroup: ONLINE, SHUNNED, OFFLINE_SOFT, and OFFLINE_HARD. Zero ONLINE means every backend in the hostgroup is in one of the three degraded states. No queries can be served for that traffic class.

Gate the PAGE condition with three requirements:

• Zero ONLINE backends sustained for more than 2 monitor intervals. With the default mysql-monitor_ping_interval of 8000ms, this formula is approximately 16 seconds. A longer debounce, such as 60 seconds, is an operational choice.
• ProxySQL_Uptime > 600 to exclude cold starts where pools are empty and the monitor has not completed a full check cycle.
• Client_Connections_non_idle > 0 OR Questions rate > 0 to exclude idle instances with no real traffic.

Without these gates, the alert fires on cold starts, idle standalone instances, and reader hostgroups that legitimately SHUN all lagging replicas during backup windows. With them, you have a confirmed service outage requiring immediate investigation.

The backends can be non-ONLINE for different reasons, each requiring a different fix. The critical first step: determine whether the backends are actually unreachable or whether ProxySQL’s own monitoring and shun logic removed them from rotation.

flowchart TD
    A["Zero ONLINE backends
in a hostgroup"] --> B{"Backends reachable
directly?"}
    B -- "No" --> C["Network partition or
backend MySQL down"]
    B -- "Yes" --> D{"Monitor checks
failing?"}
    D -- "Yes" --> E["Identify check type:
connect, ping, or lag"]
    D -- "No" --> F{"Config problem?
Check weight, drift,
Galera shunned_automatic"}
    E --> G["Connection errors: check
max_connect_errors, credentials"]
    E --> H["Lag: check replication
on all backends"]
    F --> I["Fix config and
LOAD TO RUNTIME"]

Common causes

Quick checks

All commands query the ProxySQL admin interface (port 6032) and are read-only. Safe to run during an active incident.

# Check backend status and connection pool per hostgroup
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree, ConnOK, ConnERR FROM stats_mysql_connection_pool;"

# Check runtime server configuration (weight, max_connections, status)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port, status, weight, max_connections FROM runtime_mysql_servers;"

# Check monitor check success/failure counts
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name LIKE 'MySQL_Monitor_%';"

# Verify alert gating conditions
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('ProxySQL_Uptime', 'Client_Connections_non_idle', 'Questions');"

# Check recent monitor ping log entries for failure patterns
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_ping_log ORDER BY time_start_us DESC LIMIT 30;"

# Check monitor workers are alive (zero means monitoring is disabled)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('MySQL_Monitor_Workers', 'MySQL_Monitor_Workers_Aux');"

# Direct backend connectivity check (bypass ProxySQL entirely)
mysql -u <backend_user> -p -h <backend_host> -P <backend_port> \
  -e "SELECT @@hostname, @@read_only;"

# Check per-errno error breakdown (ProxySQL 2.x+)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM stats_mysql_errors ORDER BY last_seen DESC LIMIT 20;"

How to diagnose it

1. Confirm the outage is real. Check ProxySQL_Uptime, Client_Connections_non_idle, and Questions rate. If uptime is under 600, this may be a cold start. If non_idle and Questions are both zero, the instance is idle and the zero-ONLINE state may be expected. The outage is real when uptime > 600 and there is active traffic with zero ONLINE backends.

2. Verify backends are reachable directly. Connect to at least one backend MySQL instance from the ProxySQL host, bypassing the proxy. If the direct connection fails, the problem is network, firewall, DNS, or the backend MySQL process itself. Check security groups, firewall rules, and DNS resolution. If the direct connection succeeds, ProxySQL’s monitor or shun logic is the issue.

3. Identify the degraded state. Query stats_mysql_connection_pool and look at the status column for each backend. If all are SHUNNED, the backends were removed by ProxySQL’s protection mechanism, not by operator action. If all are OFFLINE_HARD, someone or something explicitly removed them. If the hostgroup has no entries in runtime_mysql_servers, the configuration is missing servers.

4. Check monitor check results. Look at MySQL_Monitor_connect_check_ERR and MySQL_Monitor_ping_check_ERR in stats_mysql_global. Rising error counts tell you which check type is failing. Cross-reference with the monitor log tables (monitor.mysql_server_ping_log, monitor.mysql_server_connect_log) for specific error messages. If MySQL_Monitor_Workers is zero, the monitor module is disabled and status decisions are stale.

5. Check for configuration problems. Compare mysql_servers (MEMORY layer) against runtime_mysql_servers (RUNTIME layer). If they differ, a LOAD MYSQL SERVERS TO RUNTIME was never executed after a change. Check the weight column: weight=0 means the server is never used for routing despite appearing ONLINE. Check whether Galera max_writers is configured.

6. Check for backend-side blocking. If ProxySQL has been generating many failed connection attempts, MySQL’s max_connect_errors (default 100) may have blocked the ProxySQL host. The backend returns “Host ‘X’ is blocked because of many connection errors.” This creates a death spiral: failed connections trigger shunning, shun recovery triggers more retries, more failures accumulate, and the block persists.

Metrics and signals to monitor

Fixes

Backends genuinely unreachable

If direct connectivity to backends fails, the problem is outside ProxySQL. Fix the network path (firewall rules, security groups, DNS resolution) or restart the backend MySQL process. Once backends are reachable, ProxySQL’s monitor will detect recovery within one check interval and transition backends back to ONLINE automatically.

All backends SHUNNED from connection errors

The version-dependent effective shun threshold is min(mysql-shun_on_failures, mysql-connect_retries_on_failure + 1) since ProxySQL v2.5.0, and min(mysql-shun_on_failures, mysql-connect_retries_on_failure) through v2.4.x. With defaults of 5 and 10 respectively, both formulas produce 5. If all backends in a hostgroup accumulate errors faster than the recovery cycle, they stay perpetually SHUNNED.

MySQL max_connect_errors blocking. If the backend has blocked the ProxySQL host, run mysqladmin flush-hosts on the affected backend to clear the block. This is disruptive to that backend’s connection state but necessary to break the death spiral. Then investigate why the errors accumulated.

Intermittent network issues. If connections succeed sometimes and fail other times, ConnERR accumulates alongside ConnOK. Run mtr or traceroute between the ProxySQL host and backend. Consider increasing mysql-shun_recovery_time_sec (default 10s) to give the network more time to recover between retry cycles.

Raising mysql-shun_on_failures alone may have no effect if mysql-connect_retries_on_failure is lower.

All backends SHUNNED from replication lag

If all replicas exceed their configured max_replication_lag, ProxySQL correctly SHUNNs them all. The fix is on the replication side: investigate why replicas cannot keep up. Common causes include large DDL operations, long-running transactions on the primary, or insufficient replica I/O capacity.

If the lag is transient (backup window, batch job), consider downgrading the PAGE to a TICKET for reader-only hostgroups where a writer hostgroup still has ONLINE backends.

Monitor credentials expired

When mysql-monitor_password no longer matches the backend grants, all monitor checks fail. ProxySQL cannot verify backend health and shuns them. Data-plane connections using credentials from mysql_users may still work briefly from the connection pool until they drain.

Fix both credential sets if needed:

# Changes runtime state.
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SET mysql-monitor_password='<new_password>'; LOAD MYSQL VARIABLES TO RUNTIME; SAVE MYSQL VARIABLES TO DISK;"

# Changes runtime state. Disconnects active user sessions on next reconnect.
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "UPDATE mysql_users SET password='<new_password>' WHERE username='<user>'; LOAD MYSQL USERS TO RUNTIME; SAVE MYSQL USERS TO DISK;"

After updating, monitor check results should transition from ERR to OK within one check interval, and backends will return to ONLINE.

Configuration problems

Config drift. If mysql_servers and runtime_mysql_servers differ, load the pending changes. Verify the MEMORY configuration is correct before executing:

# Changes runtime state. Verify mysql_servers contents first.
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;"

weight=0. Set weight to a non-zero value for each affected server, then load to runtime. ProxySQL default weight is 1.

Galera max_writers recovery issue. Issue #3618 reproduces skipped promotion on v2.2.0 and v2.2.2 and remains open, so do not assume it is fixed in a later version without testing your deployment. If the active writer fails, compare writer/backup placement and max_writers, then promote a healthy node manually or temporarily adjust max_writers; verify zero-ONLINE clears before restoring normal alerting.

Prevention

• Track configuration layer consistency. Compare mysql_servers vs runtime_mysql_servers vs disk state regularly. Alert on divergence sustained for more than 10 minutes. Most config drift incidents happen when an operator changes MEMORY but forgets LOAD TO RUNTIME, or loads to RUNTIME but forgets SAVE TO DISK.

• Monitor the monitor. Track MySQL_Monitor_Workers and all MySQL_Monitor_*_check_ERR counters. If the monitor module is disabled or failing, ProxySQL’s status decisions become stale. A zero-ONLINE condition caused by monitor failure is preventable with early detection.

• Watch shun and unshun cycles. Track ConnERR rates per backend against the effective shun threshold. If error rates approach the threshold on multiple backends simultaneously, investigate before they all get shunned.

• Keep monitor credentials in sync. Treat mysql-monitor_password changes as a coordinated operation with backend grant updates. Credential rotation that does not update ProxySQL is one of the most common causes of silent backend shunning.

• Verify weight values when adding servers. Setting weight=0 silently excludes a server from routing. Validate weight values as part of server provisioning.

• Set replication lag thresholds thoughtfully. max_replication_lag should reflect what your application can tolerate, not a value that causes all replicas to be SHUNNED simultaneously during normal operations.

How Netdata helps

Netdata’s ProxySQL collector surfaces the specific signals that distinguish a real backend outage from a cold start, idle instance, or transient shun cycle:

• Per-second backend status tracking shows exactly when each backend transitioned from ONLINE to SHUNNED or OFFLINE_HARD, and whether the transition correlates with a monitor check failure, a ConnERR spike, or a configuration change.
• Monitor check result rates (connect, ping, read_only, replication_lag OK/ERR) provide leading indicators before backends reach zero-ONLINE. A rising ERR rate across check types signals network or credential problems before the outage is total.
• ProxySQL_Uptime and Client_Connections_non_idle gating eliminates false-positive pages from cold starts and idle instances, so the alert only fires when real traffic is failing.
• ConnERR per backend correlates directly with shun events, making it clear whether backends are failing because ProxySQL cannot reach them or because MySQL blocked the ProxySQL host.
• Questions rate and Client_Connections_aborted confirm client-visible impact and distinguish “queries are failing” from “there is no traffic to this hostgroup.”

Related guides

• ProxySQL backend SHUNNED: why a healthy backend gets pulled out of rotation
• How ProxySQL actually works in production: a mental model for operators
• ProxySQL monitoring checklist: the signals every production proxy needs
• ProxySQL monitoring maturity model: from survival to expert