$ guides / proxysql / proxysql-cpu-thread-saturation ▌

Operations Guides

ProxySQL worker thread CPU saturation: mysql-threads as a hard parallelism ceiling

When every query through ProxySQL slows down simultaneously, regardless of backend or query digest, the proxy itself is the bottleneck. The most common cause is worker thread CPU saturation: all mysql-threads worker threads are pegged at or near 100% CPU, and their epoll event loops can no longer service connections without adding queuing delay.

mysql-threads (default 4) cannot be changed at runtime. It is set at startup and caps query-processing parallelism absolutely. Adding backends or client connections does not raise this ceiling.

The diagnostic signature: latency increases uniformly across all hostgroups, digests, and backends. Backend connection pools show free connections (ConnFree > 0), meaning backends are idle and waiting for the proxy to send work. This is the opposite of backend pool exhaustion, where queries queue for a free backend connection.

What this means

Each ProxySQL worker thread runs a non-blocking epoll event loop. Client connections are distributed across worker threads at accept time and pinned to a specific thread for their lifetime. The event loop multiplexes many connections per thread: parsing queries, matching mysql_query_rules, managing backend connection multiplexing, and forwarding traffic.

When a worker thread hits 100% CPU, the event loop cannot cycle fast enough. Every connection on that thread sees increased latency because the thread takes longer between events. The result is uniform latency inflation: every query gets slower by roughly the same delta, independent of query type or backend.

With the default of 4 threads on a 16- or 32-core machine, ProxySQL uses at most 4 cores for query processing. The remaining cores sit idle even under heavy load. Aggregate process CPU might read 400% (4 threads at 100% each), which looks moderate on a 32-core box but is the hard ceiling.

flowchart TD
    A["mysql-threads = 4
hard ceiling, set at startup"] --> B["4 worker threads
each running epoll event loop"]
    B --> C["All threads near 100% CPU
under heavy load"]
    C --> D["Event loop latency rises"]
    D --> E["Uniform query slowdown
across all backends and digests"]
    E --> F["Backends show ConnFree > 0
idle, waiting for the proxy"]

Common causes

Cause	What it looks like	First thing to check
Insufficient mysql-threads for workload	Process CPU plateaus at N x 100% where N = mysql-threads; backends idle with free connections	`SELECT variable_name, variable_value FROM global_variables WHERE variable_name = 'mysql-threads';`
Complex regex in query rules	`Query_Processor_time_nsec` grows disproportionate to Questions rate	`SELECT variable_name, variable_value FROM stats_mysql_global WHERE variable_name = 'Query_Processor_time_nsec';`
TLS termination overhead	One or two threads pegged while others are moderate; correlates with connection churn	Check `use_ssl` in `mysql_servers` and client TLS settings
mysql-threads set too high (above ~16)	High system CPU (%sy), lower throughput despite low user CPU	Compare throughput before and after thread count change
Excessive idle-connection polling	CPU high even when query rate is low; many client connections	Check whether `--idle-threads` is active on the ProxySQL process

Quick checks

# Throughput and uptime
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM stats_mysql_global WHERE variable_name IN ('Questions','ProxySQL_Uptime');"

# Per-thread CPU: the critical check
ps -L -p $(pidof proxysql) -o pid,lwp,pcpu

# Per-thread CPU over time (1-second intervals)
pidstat -p $(pidof proxysql) -t 1

# Query Processor time: should grow proportionally with Questions rate
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM stats_mysql_global WHERE variable_name IN ('Query_Processor_time_nsec','Questions','Slow_queries');"

# Confirm backends are idle (ConnFree > 0 means the proxy, not backends, is the bottleneck)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree FROM stats_mysql_connection_pool;"

# Current mysql-threads value
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM global_variables WHERE variable_name = 'mysql-threads';"

How to diagnose it

Confirm uniform latency inflation. Check stats_mysql_commands_counters for a histogram shift toward higher latency buckets across all command types, not just SELECT or INSERT. If only one command type degrades, the problem is likely backend-specific.
Check per-thread CPU. Run ps -L -p $(pidof proxysql) -o pid,lwp,pcpu or pidstat -p $(pidof proxysql) -t 1. If all worker threads are near 100% CPU, you have thread saturation. If only one or two threads are pegged while others are idle, suspect connection imbalance, TLS overhead concentrated on specific threads, or a hot regex rule.
Verify backends are not the bottleneck. Query stats_mysql_connection_pool and confirm ConnFree > 0 across backends. If ConnFree is zero and ConnUsed is high, the problem is backend pool exhaustion, not thread saturation. See backend connection pool exhaustion.
Check Query_Processor_time_nsec trend. This counter measures time spent inside the Query Processor module: parsing, rule matching, and digest calculation. Take two readings a few seconds apart. If the rate of growth exceeds the rate of growth in Questions, complex query rules are consuming disproportionate CPU.
Distinguish proxy overhead from backend latency. If backend ping latency (Latency_us in stats_mysql_connection_pool) is stable but client-perceived latency is increasing, the delta is proxy overhead. Worker thread saturation adds queuing delay inside the event loop, not backend execution time.
Compare mysql-threads against core count. If mysql-threads is 4 on a machine with 16 or more cores and CPU is saturating, increase the thread count. This requires a restart.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Per-thread CPU (`ps -L`)	Shows individual worker thread utilization, not just process aggregate	Any thread consistently above 90% CPU
Process CPU percentage	Quick check against the hard ceiling of `mysql-threads * 100%`	Sustained above 60% of the ceiling (e.g., >240% with mysql-threads=4)
`Query_Processor_time_nsec`	Time spent parsing and rule-matching; isolates proxy CPU overhead from backend wait	Growing faster than Questions rate
`Questions` rate	Throughput baseline; correlate with CPU to detect CPU-bound periods	Drop in Questions rate while CPU stays high
Backend `ConnFree`	Free backend connections prove the proxy, not the backend pool, is the bottleneck	`ConnFree > 0` across all backends while latency rises
`stats_mysql_commands_counters` histogram	Latency distribution across command types; uniform shift indicates proxy overhead	All command types shift to higher buckets simultaneously
Multiplexing ratio	High connection count with multiplexing disabled increases per-connection CPU overhead	Ratio trending toward 1:1
`MySQL_Thread_Workers`	Confirms the configured MySQL traffic worker count in `stats_mysql_global`	Value does not match the intended `mysql-threads` after a restart

Fixes

Increase mysql-threads (requires restart)

The primary fix when worker threads are genuinely saturated. Set mysql-threads close to the number of available CPU cores.

-- Set the new thread count (goes to MEMORY layer)
SET mysql-threads=16;

-- Persist to disk (LOAD TO RUNTIME will fail for this variable)
SAVE MYSQL VARIABLES TO DISK;

Then restart ProxySQL. Official documentation identifies mysql-threads and mysql-interfaces as not dynamic, and mysql-stacksize as having no runtime effect; mysql-have_ssl is dynamic.

# Disruptive: drops all active client connections
systemctl restart proxysql

Do not set mysql-threads excessively high. On high-core machines, high thread counts can reduce throughput through excessive context switching, with system CPU (%sy) rising while throughput drops. The optimum depends on workload, connection count, and kernel behavior; benchmark changes on representative production traffic before deploying large increases.

Optimize query rules

If Query_Processor_time_nsec is elevated relative to the Questions rate, complex regex patterns in mysql_query_rules are consuming CPU. Every incoming query is matched against the rule chain sequentially, and expensive regex evaluation runs per query.

-- Review active rules for complex patterns
SELECT rule_id, match_digest, match_pattern, destination_hostgroup, apply
FROM runtime_mysql_query_rules
WHERE active=1
ORDER BY rule_id;

Emergency mitigation if a specific rule is the culprit:

UPDATE mysql_query_rules SET active=0 WHERE rule_id=<problem_rule>;
LOAD MYSQL QUERY RULES TO RUNTIME;

Unlike mysql-threads, query rule changes take effect at runtime without a restart.

Scale out instead of up

When a single ProxySQL instance cannot handle the load even with an optimized thread count, add more ProxySQL instances behind a load balancer. Each instance gets its own mysql-threads worker pool, and aggregate parallelism scales linearly. This is preferable to pushing mysql-threads above ~16 on a single instance.

Reduce TLS overhead

If TLS termination is consuming worker thread CPU (visible as one or two threads disproportionately pegged while connection churn is high), consider terminating TLS at a load balancer or sidecar proxy instead of at ProxySQL itself. Official ProxySQL documentation does not document a configuration for TLS session resumption or session tickets; reduce connection churn instead and verify TLS behavior against your deployed version.

Prevention

Set mysql-threads proactively. Match thread count to available cores at deployment time, capped at approximately 16 unless testing proves higher values help. Do not wait for saturation.
Monitor per-thread CPU, not just process aggregate. Process-level CPU percentage hides individual thread saturation. Alert on per-thread CPU from ps -L or pidstat -t.
Track Query_Processor_time_nsec as a rate. If it grows faster than the Questions rate, query rule complexity is increasing. Catch this before it saturates threads.
Keep peak process CPU below 60%. Above this threshold, event loop latency begins affecting every query. The headroom absorbs traffic spikes and connection storms.
Do not add backends to fix proxy-side bottlenecks. Adding more MySQL backends does not help when the proxy itself is CPU-saturated. The proxy cannot send queries to backends faster than its worker threads can process them.
Review query rules during capacity planning. Complex regex patterns are a per-query CPU tax. Simplify rules, reorder to put high-traffic matches early, and ensure apply=1 terminates evaluation on the hot path.

How Netdata helps

Per-second CPU metrics with per-core and per-process breakdowns show worker thread saturation developing in real time and distinguish ProxySQL process saturation from system-wide CPU contention caused by co-located workloads.
Correlating ProxySQL stats (Questions, Query_Processor_time_nsec) with host-level CPU confirms whether latency spikes align with CPU saturation events, making it immediately clear whether the proxy or the backend is the bottleneck.
Backend pool metrics displayed next to CPU let you confirm in one view that backends are idle while the proxy is saturated, the definitive signature of thread starvation.
Anomaly detection on CPU and latency patterns surfaces gradual degradation before it crosses hard thresholds, critical for a variable that requires a restart to change.