$ guides / proxysql / proxysql-monitor-workers-zero ▌

Operations Guides

ProxySQL MySQL_Monitor_Workers is zero: health checks stopped and status is stale

When MySQL_Monitor_Workers in stats_mysql_global reads zero and stays there, ProxySQL’s monitor module has stopped probing backends. No connect checks, no ping checks, no read-only checks, no replication lag checks. The status values in runtime_mysql_servers and stats_mysql_connection_pool are frozen at whatever they were when the last check ran. A backend that crashed five minutes ago still shows ONLINE.

This state is silent. The data plane keeps routing queries. Nothing crashes, nothing logs an error visible to most dashboards. The only outward signal is that ProxySQL stops reacting to real backend health changes: a writer failover goes undetected, a lagging replica stays in rotation, a dead backend keeps receiving queries until clients time out.

The primary diagnostic signal is MySQL_Monitor_Workers == 0 sustained over multiple polling intervals. Correlate with flatlined monitor check counters (MySQL_Monitor_connect_check_OK, MySQL_Monitor_ping_check_OK, and their ERR counterparts). If neither OK nor ERR counters increment, no checks are running. The monitor log tables (monitor.mysql_server_connect_log, monitor.mysql_server_ping_log, monitor.mysql_server_read_only_log) also stop receiving new rows.

What this means

ProxySQL’s monitor module runs on dedicated background threads separate from the worker threads that handle client traffic. These threads probe backends and feed results into the hostgroup manager, which transitions backends between ONLINE, SHUNNED, OFFLINE_SOFT, and OFFLINE_HARD.

With zero active workers, the hostgroup manager retains whatever status decisions it last made. A backend that was ONLINE stays ONLINE regardless of actual health. A backend that was SHUNNED stays SHUNNED even after recovery.

Under normal operation, MySQL_Monitor_Workers should be at least mysql-monitor_threads_min (default 8 in ProxySQL v2.0+). The companion counter MySQL_Monitor_Workers_Started tracks cumulative monitor thread starts since process start. If Workers_Started is greater than zero but Workers is zero, threads were started and then exited without replacement.

The severity classification for this condition is TICKET, not PAGE. Monitor failure is a leading indicator: actual impact (stale routing, queries hitting dead backends) is downstream. The longer this persists, the more likely real backend failures go undetected and cause user-facing impact.

flowchart TD
    A["MySQL_Monitor_Workers == 0"] --> B{"mysql-monitor_enabled = true?"}
    B -- "No" --> C["Set true, LOAD TO RUNTIME"]
    B -- "Yes" --> D{"Started with -M flag?"}
    D -- "Yes" --> E["Restart without -M"]
    D -- "No" --> F{"Workers_Started > 0?"}
    F -- "Yes" --> G["Silent thread death. Check logs, restart."]
    F -- "No" --> H["Never started. Check config and init."]

Common causes

Cause	What it looks like	First thing to check
Started with `-M` or `--no-monitor` flag	Workers zero since process start; never increments; `Workers_Started` also zero	Process command line for the `-M` flag
`mysql-monitor_enabled` set to false	Workers zero; all other monitor variables present and configurable but module inactive	`global_variables` table for `mysql-monitor_enabled`
Silent monitor thread death	Workers were nonzero, then dropped to zero; `Workers_Started > 0`; check counters flatline from that point	ProxySQL error log for assertions or errors around the time workers dropped

Quick checks

All read-only queries against the admin interface (default port 6032) or the host OS. None modify ProxySQL state.

# Check current monitor worker count and related counters
mysql -u admin -padmin -h 127.0.0.1 -P 6032 -e "
  SELECT Variable_Name, Variable_Value
  FROM stats_mysql_global
  WHERE Variable_Name IN (
    'MySQL_Monitor_Workers',
    'MySQL_Monitor_Workers_Aux',
    'MySQL_Monitor_Workers_Started'
  );"

# Verify monitor is enabled at runtime
mysql -u admin -padmin -h 127.0.0.1 -P 6032 -e "
  SELECT variable_name, variable_value
  FROM global_variables
  WHERE variable_name IN (
    'mysql-monitor_enabled',
    'mysql-monitor_threads_min',
    'mysql-monitor_threads_max'
  );"

# Check if monitor check counters are incrementing (run twice, several seconds apart)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 -e "
  SELECT Variable_Name, Variable_Value
  FROM stats_mysql_global
  WHERE Variable_Name LIKE 'MySQL_Monitor_%check%';"

# Check freshness of monitor log tables (last entry timestamp)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 -e "
  SELECT * FROM monitor.mysql_server_ping_log
  ORDER BY time_start_us DESC LIMIT 5;"

# Check if ProxySQL was started with -M (--no-monitor). If multiple instances,
# replace $(pidof proxysql) with the specific PID.
cat /proc/$(pidof proxysql)/cmdline | tr '\0' ' '

# Check ProxySQL error log for watchdog assertions or monitor errors
# Log path varies by installation; adjust if different
grep -iE 'monitor|watchdog|assert' /var/lib/proxysql/proxysql.log | tail -30

# Verify backend status values (compare against direct backend checks to detect staleness)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 -e "
  SELECT hostgroup, srv_host, srv_port, status
  FROM stats_mysql_connection_pool;"

How to diagnose it

Confirm MySQL_Monitor_Workers is zero and sustained. Poll twice, 10-15 seconds apart. A single zero reading can be a transient artifact or cold-start condition during the first few seconds after process start.
Check mysql-monitor_enabled. If false, the monitor module was disabled at runtime or in configuration. Simplest cause to fix.
Check the process command line for -M. If ProxySQL was started with -M or --no-monitor, the monitor is disabled at the process level and cannot be re-enabled by runtime variable changes.
Check MySQL_Monitor_Workers_Started. If zero and the process was not started with -M, monitor threads were never created. This points to a configuration or initialization problem. If greater than zero, threads started and then exited without replacement.
Verify monitor check counters are flatlined. Query the MySQL_Monitor_*_check_OK and MySQL_Monitor_*_check_ERR counters twice with a gap. If neither OK nor ERR increments across any check type (connect, ping, read_only, replication_lag), the monitor is definitively not running checks.
Check monitor log table freshness. Query monitor.mysql_server_ping_log ordered by time_start_us DESC. If the most recent entry is significantly older than your configured mysql-monitor_ping_interval, no new checks are being recorded. The default mysql-monitor_ping_interval is 8000 ms.
Check the ProxySQL error log. Look for watchdog assertions, thread crash messages, or monitor errors around the time workers dropped to zero.
Check for known bug patterns. GitHub issue #2860 documents a case in ProxySQL 2.0.4 where all monitor checks stopped with mysql-monitor_enabled=true and no error logged. A restart restored checks in that report, but the issue records no fixed release; do not assume upgrading alone removes this failure mode.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`MySQL_Monitor_Workers`	Direct measure of active monitor threads	Zero sustained over multiple poll intervals
`MySQL_Monitor_Workers_Started`	Cumulative thread starts; distinguishes “never started” from “started then exited”	Greater than zero while `Workers` is zero
Monitor check OK/ERR counters per type	Proves checks are actually executing	Both OK and ERR flatlined (no increments)
Monitor log table freshness (`time_start_us`)	Independent confirmation that checks are being recorded	Most recent entry older than configured check interval
Backend status in `stats_mysql_connection_pool`	Shows whether stale decisions are causing routing problems	Status values unchanged for abnormally long periods relative to check intervals
ProxySQL error log	May contain watchdog assertions or monitor crash traces	New error entries around the time workers dropped
`ProxySQL_Uptime`	Context for distinguishing cold start from runtime failure	Low uptime with zero workers may be transient initialization

Fixes

ProxySQL started with `-M` or `--no-monitor`

If the process command line includes -M, the monitor module is disabled at startup and no runtime variable change can re-enable it. You must restart ProxySQL without the flag.

Check your service definition (systemd unit, init script, container command) for the -M flag, remove it, and restart. This restart drops all client connections and resets all stats tables. Schedule it as a maintenance event if possible.

This flag is sometimes added intentionally for deployments where an external health-check mechanism replaces ProxySQL’s built-in monitor. If that is your architecture, MySQL_Monitor_Workers == 0 is expected and not a problem.

`mysql-monitor_enabled` set to false

Re-enable the monitor:

SET mysql-monitor_enabled = true;
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

LOAD MYSQL VARIABLES TO RUNTIME activates the change immediately. SAVE MYSQL VARIABLES TO DISK persists it across restarts. After loading to runtime, monitor workers should start within seconds. Verify by checking MySQL_Monitor_Workers again after 5-10 seconds.

If someone set this to false intentionally (for example, during maintenance or troubleshooting), confirm with your team before re-enabling.

Silent monitor thread death

If mysql-monitor_enabled is true, the process was not started with -M, and workers are still zero, the monitor threads have exited without being replaced. This is the pattern documented in GitHub issue #2860.

Collect diagnostic data before restarting. Error log entries, MySQL_Monitor_Workers_Started value, monitor variable settings, and ProxySQL version. This evidence will be lost after restart.
Check whether a ProxySQL upgrade is available. The root cause of issue #2860 was never identified, but later versions may include fixes for monitor thread lifecycle bugs.
Restart ProxySQL. This is the known workaround. It drops all client connections and resets all stats tables.

# WARNING: drops all client connections and resets stats tables
sudo systemctl restart proxysql

After restart, verify that MySQL_Monitor_Workers returns to a nonzero value (at least mysql-monitor_threads_min, default 8) and that check counters begin incrementing within the first monitor interval.

Prevention

Alert on MySQL_Monitor_Workers == 0 sustained. Use a duration gate of at least 2 polling intervals to avoid false positives during cold start. Exclude the first 30-60 seconds after process start using ProxySQL_Uptime.
Correlate with monitor check counter rates. A flatlined OK rate (zero increments across all check types) confirms the monitor is not running, even if the worker counter is stale or reporting incorrectly.
Monitor MySQL_Monitor_Workers_Started for unexpected patterns. If zero on a running process not started with -M, the monitor never initialized. Investigate before traffic patterns mask the problem.
Review startup flags in configuration management. Ensure -M is not present in production deployment templates unless the monitor is intentionally disabled by design.
Keep ProxySQL updated. Silent monitor thread death is a known bug pattern. Running the latest stable release reduces the chance of encountering unfixed thread lifecycle issues.
Centralize ProxySQL error logs. Watchdog assertions and monitor errors may only appear in the ProxySQL log file, not in stats tables. Persist log collection so diagnostic evidence survives restarts.

Correlating with Netdata

If ProxySQL is monitored via Netdata, several signals help confirm and scope the problem:

MySQL_Monitor_Workers per-second tracking shows the exact moment workers dropped to zero, not just the next polling cycle.
Monitor check counter rates (OK and ERR per check type) provide independent confirmation. When rates flatline across connect, ping, read_only, and replication_lag checks simultaneously, the monitor is definitively stopped.
Backend status in stats_mysql_connection_pool shows whether stale decisions are affecting routing. Correlating the worker drop with backend status freezing confirms downstream impact.
Anomaly detection on monitor check rates can flag the flatlining pattern before an explicit zero-threshold alert fires.
Correlation views let you overlay the worker drop against client connection aborts, backend connection errors, and query latency changes that follow stale routing decisions.