When every query through ProxySQL slows down simultaneously, regardless of backend or query digest, the proxy itself is the bottleneck. The most common cause is worker thread CPU saturation: all mysql-threads worker threads are pegged at or near 100% CPU, and their epoll event loops can no longer service connections without adding queuing delay.

mysql-threads (default 4) cannot be changed at runtime. It is set at startup and caps query-processing parallelism absolutely. Adding backends or client connections does not raise this ceiling.

The diagnostic signature: latency increases uniformly across all hostgroups, digests, and backends. Backend connection pools show free connections (ConnFree > 0), meaning backends are idle and waiting for the proxy to send work. This is the opposite of backend pool exhaustion, where queries queue for a free backend connection.

What this means

Each ProxySQL worker thread runs a non-blocking epoll event loop. Client connections are distributed across worker threads at accept time and pinned to a specific thread for their lifetime. The event loop multiplexes many connections per thread: parsing queries, matching mysql_query_rules, managing backend connection multiplexing, and forwarding traffic.

When a worker thread hits 100% CPU, the event loop cannot cycle fast enough. Every connection on that thread sees increased latency because the thread takes longer between events. The result is uniform latency inflation: every query gets slower by roughly the same delta, independent of query type or backend.

With the default of 4 threads on a 16- or 32-core machine, ProxySQL uses at most 4 cores for query processing. The remaining cores sit idle even under heavy load. Aggregate process CPU might read 400% (4 threads at 100% each), which looks moderate on a 32-core box but is the hard ceiling.

flowchart TD
    A["mysql-threads = 4
hard ceiling, set at startup"] --> B["4 worker threads
each running epoll event loop"]
    B --> C["All threads near 100% CPU
under heavy load"]
    C --> D["Event loop latency rises"]
    D --> E["Uniform query slowdown
across all backends and digests"]
    E --> F["Backends show ConnFree > 0
idle, waiting for the proxy"]

Common causes

CauseWhat it looks likeFirst thing to check
Insufficient mysql-threads for workloadProcess CPU plateaus at N x 100% where N = mysql-threads; backends idle with free connectionsSELECT variable_name, variable_value FROM global_variables WHERE variable_name = 'mysql-threads';
Complex regex in query rulesQuery_Processor_time_nsec grows disproportionate to Questions rateSELECT variable_name, variable_value FROM stats_mysql_global WHERE variable_name = 'Query_Processor_time_nsec';
TLS termination overheadOne or two threads pegged while others are moderate; correlates with connection churnCheck use_ssl in mysql_servers and client TLS settings
mysql-threads set too high (above ~16)High system CPU (%sy), lower throughput despite low user CPUCompare throughput before and after thread count change
Excessive idle-connection polling (pre-v2.0.11)CPU high even when query rate is low; many client connectionsCheck ProxySQL version and whether --idle-threads is active

Quick checks

# Throughput and uptime
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM stats_mysql_global WHERE variable_name IN ('Questions','ProxySQL_Uptime');"

# Per-thread CPU: the critical check
ps -L -p $(pidof proxysql) -o pid,lwp,pcpu

# Per-thread CPU over time (1-second intervals)
pidstat -p $(pidof proxysql) -t 1

# Query Processor time: should grow proportionally with Questions rate
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM stats_mysql_global WHERE variable_name IN ('Query_Processor_time_nsec','Questions','Slow_queries');"

# Confirm backends are idle (ConnFree > 0 means the proxy, not backends, is the bottleneck)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree FROM stats_mysql_connection_pool;"

# Current mysql-threads value
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM global_variables WHERE variable_name = 'mysql-threads';"

How to diagnose it

  1. Confirm uniform latency inflation. Check stats_mysql_commands_counters for a histogram shift toward higher latency buckets across all command types, not just SELECT or INSERT. If only one command type degrades, the problem is likely backend-specific.

  2. Check per-thread CPU. Run ps -L -p $(pidof proxysql) -o pid,lwp,pcpu or pidstat -p $(pidof proxysql) -t 1. If all worker threads are near 100% CPU, you have thread saturation. If only one or two threads are pegged while others are idle, suspect connection imbalance, TLS overhead concentrated on specific threads, or a hot regex rule.

  3. Verify backends are not the bottleneck. Query stats_mysql_connection_pool and confirm ConnFree > 0 across backends. If ConnFree is zero and ConnUsed is high, the problem is backend pool exhaustion, not thread saturation. See backend connection pool exhaustion.

  4. Check Query_Processor_time_nsec trend. This counter measures time spent inside the Query Processor module: parsing, rule matching, and digest calculation. Take two readings a few seconds apart. If the rate of growth exceeds the rate of growth in Questions, complex query rules are consuming disproportionate CPU.

  5. Distinguish proxy overhead from backend latency. If backend ping latency (Latency_us in stats_mysql_connection_pool) is stable but client-perceived latency is increasing, the delta is proxy overhead. Worker thread saturation adds queuing delay inside the event loop, not backend execution time.

  6. Compare mysql-threads against core count. If mysql-threads is 4 on a machine with 16 or more cores and CPU is saturating, increase the thread count. This requires a restart.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Per-thread CPU (ps -L)Shows individual worker thread utilization, not just process aggregateAny thread consistently above 90% CPU
Process CPU percentageQuick check against the hard ceiling of mysql-threads * 100%Sustained above 60% of the ceiling (e.g., >240% with mysql-threads=4)
Query_Processor_time_nsecTime spent parsing and rule-matching; isolates proxy CPU overhead from backend waitGrowing faster than Questions rate
Questions rateThroughput baseline; correlate with CPU to detect CPU-bound periodsDrop in Questions rate while CPU stays high
Backend ConnFreeFree backend connections prove the proxy, not the backend pool, is the bottleneckConnFree > 0 across all backends while latency rises
stats_mysql_commands_counters histogramLatency distribution across command types; uniform shift indicates proxy overheadAll command types shift to higher buckets simultaneously
Multiplexing ratioHigh connection count with multiplexing disabled increases per-connection CPU overheadRatio trending toward 1:1

Fixes

Increase mysql-threads (requires restart)

The primary fix when worker threads are genuinely saturated. Set mysql-threads close to the number of available CPU cores.

-- Set the new thread count (goes to MEMORY layer)
SET mysql-threads=16;

-- Persist to disk (LOAD TO RUNTIME will fail for this variable)
SAVE MYSQL VARIABLES TO DISK;

Then restart ProxySQL. The restart is mandatory because mysql-threads is one of the ProxySQL variables that cannot be changed at runtime, alongside mysql-interfaces and mysql-stacksize.

# Disruptive: drops all active client connections
systemctl restart proxysql

Do not set mysql-threads excessively high. On high-core machines, setting mysql-threads above approximately 16 can reduce throughput because each worker reacts faster to network events but performs less work per event cycle, causing excessive context switching. System CPU (%sy) rises while throughput drops. Test before deploying large increases.

Optimize query rules

If Query_Processor_time_nsec is elevated relative to the Questions rate, complex regex patterns in mysql_query_rules are consuming CPU. Every incoming query is matched against the rule chain sequentially, and expensive regex evaluation runs per query.

-- Review active rules for complex patterns
SELECT rule_id, match_digest, match_pattern, destination_hostgroup, apply
FROM runtime_mysql_query_rules
WHERE active=1
ORDER BY rule_id;

Emergency mitigation if a specific rule is the culprit:

UPDATE mysql_query_rules SET active=0 WHERE rule_id=<problem_rule>;
LOAD MYSQL QUERY RULES TO RUNTIME;

Unlike mysql-threads, query rule changes take effect at runtime without a restart.

Scale out instead of up

When a single ProxySQL instance cannot handle the load even with an optimized thread count, add more ProxySQL instances behind a load balancer. Each instance gets its own mysql-threads worker pool, and aggregate parallelism scales linearly. This is preferable to pushing mysql-threads above ~16 on a single instance.

Reduce TLS overhead

If TLS termination is consuming worker thread CPU (visible as one or two threads disproportionately pegged while connection churn is high), consider terminating TLS at a load balancer or sidecar proxy instead of at ProxySQL itself. Reducing connection churn by tuning application connection pool idle and lifetime settings can also help.

Prevention

  • Set mysql-threads proactively. Match thread count to available cores at deployment time, capped at approximately 16 unless testing proves higher values help. Do not wait for saturation.
  • Monitor per-thread CPU, not just process aggregate. Process-level CPU percentage hides individual thread saturation. Alert on per-thread CPU from ps -L or pidstat -t.
  • Track Query_Processor_time_nsec as a rate. If it grows faster than the Questions rate, query rule complexity is increasing. Catch this before it saturates threads.
  • Keep peak process CPU below 60%. Above this threshold, event loop latency begins affecting every query. The headroom absorbs traffic spikes and connection storms.
  • Do not add backends to fix proxy-side bottlenecks. Adding more MySQL backends does not help when the proxy itself is CPU-saturated. The proxy cannot send queries to backends faster than its worker threads can process them.
  • Review query rules during capacity planning. Complex regex patterns are a per-query CPU tax. Simplify rules, reorder to put high-traffic matches early, and ensure apply=1 terminates evaluation on the hot path.

How Netdata helps

  • Per-second CPU metrics with per-core and per-process breakdowns show worker thread saturation developing in real time and distinguish ProxySQL process saturation from system-wide CPU contention caused by co-located workloads.
  • Correlating ProxySQL stats (Questions, Query_Processor_time_nsec) with host-level CPU confirms whether latency spikes align with CPU saturation events, making it immediately clear whether the proxy or the backend is the bottleneck.
  • Backend pool metrics displayed next to CPU let you confirm in one view that backends are idle while the proxy is saturated, the definitive signature of thread starvation.
  • Anomaly detection on CPU and latency patterns surfaces gradual degradation before it crosses hard thresholds, critical for a variable that requires a restart to change.