$ guides / proxysql / proxysql-memory-growth-oom ▌

Operations Guides

ProxySQL memory growth and OOM: what drives RSS and how to tell a leak from load

When ProxySQL RSS climbs toward system limits, the first question is whether growth is load-driven (more connections, cached result sets, or query digests) or a genuine leak. The fixes are different: load-driven growth requires tuning buffer sizes, cache limits, or digest configuration. A leak requires identifying the subsystem that is not releasing memory and possibly upgrading to a patched version.

This guide covers the memory consumers inside ProxySQL, the admin-table queries that break down where bytes are going, and the correlation tests that separate a leak from load.

ProxySQL memory model

ProxySQL process memory is a collection of subsystems, each with its own allocation pattern. The stats_memory_metrics table in the admin interface breaks these down.

The most important field is jemalloc_resident: the physical memory footprint of the ProxySQL heap as reported by jemalloc. It is the most direct measure of physical RAM consumed.

The difference between jemalloc_active and jemalloc_allocated measures fragmentation. Active memory is what jemalloc has carved out from the system; allocated is what ProxySQL has actually requested. A large and growing gap means the allocator is holding pages it cannot return to the OS, even though ProxySQL is not actively using them.

jemalloc_retained represents virtual memory jemalloc has released for reuse but kept rather than returned to the OS. ProxySQL reports the allocator’s own retained gauge, so retained can exceed resident; this is allocator retention behavior, not a leak. If it is a problem in your environment, start ProxySQL with MALLOC_CONF="retain:false" to disable retention.

The diagnostic question is always: which field in stats_memory_metrics is growing, and does that growth correlate with workload?

flowchart TD
    A["RSS growing toward limit"] --> B{"Correlated with\nClient_Connections_connected?"}
    B -->|"Yes"| C["Load-driven: check\nbuffer and cache sizing"]
    B -->|"No"| D{"query_digest_memory\nis largest field?"}
    D -->|"Yes"| E["Digest accumulation:\nTRUNCATE or tune limits"]
    D -->|"No"| F{"Query_Cache_Memory_bytes\nnear cache size limit?"}
    F -->|"Yes"| G["Cache pressure:\nreduce TTL or flush"]
    F -->|"No"| H{"jemalloc_active minus\njemalloc_allocated large?"}
    H -->|"Yes"| I["Fragmentation:\nconsider MALLOC_CONF"]
    H -->|"No"| J["Possible leak:\ncheck version, enable profiling"]

Common causes

Cause	What it looks like	First thing to check
Query digest accumulation	`query_digest_memory` is the largest field in `stats_memory_metrics`, growing steadily without bound	`SELECT query_digest_memory, SQLite3_memory_bytes, Auth_memory FROM stats_memory_metrics;`
Query cache at capacity	`Query_Cache_Memory_bytes` near `mysql-query_cache_size_MB`, high `Query_Cache_Purged` rate, declining hit ratio	`SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name LIKE 'Query_Cache%';`
Per-connection buffers under high connection count	RSS scales with `Client_Connections_connected`, drops when connections close	Correlate RSS trend with connection count trend over time
Prepared statement metadata	`Stmt_Cached` high, multiplexing disabled on many sessions	`SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name LIKE 'Stmt%';`
jemalloc fragmentation or retention	`jemalloc_active` far exceeds `jemalloc_allocated`, or `jemalloc_retained` far exceeds `jemalloc_resident`	`SELECT jemalloc_active, jemalloc_allocated, jemalloc_resident, jemalloc_retained FROM stats_memory_metrics;`
Genuine memory leak (version-specific bug)	RSS grows monotonically regardless of load, never stabilizes after workload drops	Check ProxySQL version against known leak fixes

Quick checks

Run these read-only queries against the admin interface (default port 6032) and the host OS.

# Check OS-level RSS, virtual size, and swap
cat /proc/$(pidof proxysql)/status | grep -E 'VmRSS|VmSize|VmSwap'

-- Full memory breakdown by subsystem
SELECT * FROM stats_memory_metrics;

-- Query cache memory, entries, and eviction pressure
SELECT Variable_Name, Variable_Value FROM stats_mysql_global
WHERE Variable_Name LIKE 'Query_Cache%';

-- Current connection count (correlate with RSS trend)
SELECT Variable_Name, Variable_Value FROM stats_mysql_global
WHERE Variable_Name IN (
  'Client_Connections_connected',
  'Server_Connections_connected',
  'Active_Transactions',
  'Client_Connections_hostgroup_locked'
);

-- Prepared statement cache size
SELECT Variable_Name, Variable_Value FROM stats_mysql_global
WHERE Variable_Name LIKE 'Stmt%';

What to look for immediately:

VmSwap > 0: ProxySQL is already swapping. Performance has degraded severely. Treat this as an active incident, not a tuning task.
query_digest_memory dominating: The digest table is accumulating entries and is likely the largest single consumer.
RSS growing while Client_Connections_connected is flat: Strong signal of a possible leak rather than load-driven growth.
jemalloc_retained much larger than jemalloc_resident: Allocator retention behavior. Not a leak, but it inflates the apparent footprint if you only look at /proc VmRSS.

How to diagnose it

1. Capture a baseline.

Take two snapshots of stats_memory_metrics spaced 5 to 10 minutes apart during normal operation. Record jemalloc_resident, query_digest_memory, Query_Cache_Memory_bytes (from stats_mysql_global), and Client_Connections_connected. You need deltas, not single-point readings.

2. Correlate RSS with connection count.

If jemalloc_resident grows proportionally with Client_Connections_connected, the growth is load-driven. Per-connection buffers scale with active connections. Check whether multiplexing has collapsed: if Client_Connections_hostgroup_locked is a high fraction of Client_Connections_connected, each client is pinning a backend connection, and buffer memory accumulates faster than expected.

If RSS grows while connection count is flat or declining, the growth is not load-driven. Proceed to step 3.

3. Identify the largest growing field.

Compare your two stats_memory_metrics snapshots field by field. The field with the largest delta is your primary consumer. The usual suspects:

query_digest_memory: The query digest table grows without bound. Each unique normalized query pattern adds an entry. High-cardinality workloads (many distinct query shapes, or queries with many distinct literal patterns before normalization) accelerate this.
SQLite3_memory_bytes: The internal SQLite engine backing the admin interface tables. Under normal operation this is modest, but it can grow if stats tables accumulate rows.
Auth_memory: Authentication metadata. Usually stable unless credential sets are very large.
mysql_query_rules_memory: Query rule storage. Grows with the number and complexity of rules.

4. Check query cache pressure separately.

Query cache memory is tracked in stats_mysql_global, not stats_memory_metrics. Pull Query_Cache_Memory_bytes and compare against mysql-query_cache_size_MB (default 256 MB). The limit is soft: ProxySQL uses it as a target for the purging thread, not a hard ceiling. If Query_Cache_Memory_bytes is at the limit and Query_Cache_Purged is climbing while the hit ratio (Query_Cache_count_GET_OK / Query_Cache_count_GET) is declining, the cache is churning and consuming memory without delivering value.

5. Assess fragmentation.

Calculate jemalloc_active - jemalloc_allocated from your snapshots. If this gap is large relative to jemalloc_allocated and growing, fragmentation is driving RSS up without corresponding useful allocation. This is common after bursty workload patterns that allocate and free many small objects of varying sizes.

6. Rule out version-specific leaks.

If none of the above explains the growth pattern (RSS grows monotonically, uncorrelated with any workload metric, fragmentation is not the cause), check whether your ProxySQL version has a known leak. Several leak fixes have landed across recent releases.

Official release notes record fixes such as the 2.7.2 SSL cache leak for auxiliary threads (#4801), the 3.0.2 stats_mysql_global and query-processor deletion leaks (#5006 and #5037), and the 2.5.5 SSL CA-certificate descriptor leak (#4307). Match the symptom and release notes before treating an upgrade as the resolution.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`jemalloc_resident`	Real physical footprint of the ProxySQL heap	Sustained upward trend toward system memory limit
`jemalloc_active - jemalloc_allocated`	Allocator fragmentation	Large and growing gap independent of workload
`jemalloc_retained`	Memory freed by allocator but not returned to OS	Far exceeds `jemalloc_resident` (normal but inflates footprint)
`query_digest_memory`	Query digest table size	Largest field in `stats_memory_metrics`, growing monotonically
`Query_Cache_Memory_bytes`	Cache memory usage	At or near `mysql-query_cache_size_MB` ceiling
`Query_Cache_Purged` rate	Eviction pressure	High rate with simultaneously declining hit ratio
`Client_Connections_connected`	Workload correlation	RSS growth tracking this metric indicates load-driven growth
`Stmt_Cached`	Prepared statement cache size	High count with multiplexing disabled on many sessions
VmSwap (from `/proc/<pid>/status`)	Swap usage	Any value greater than zero means degraded performance

Fixes

Query digest accumulation

The most immediate relief is to reset the digest table. This is non-blocking and safe:

-- Reads and clears the digest table in one operation
SELECT * FROM stats_mysql_query_digest_reset;

-- Alternative: truncate without reading (loses current stats)
TRUNCATE TABLE stats_mysql_query_digest;

For longer-term control, tune the digest configuration variables. Load them to runtime and save to disk after changing:

mysql-query_digests_normalize_digest_text (default false): Set to true to deduplicate digest text across schemas. This can reduce memory when the same query pattern runs against multiple schemas.
mysql-query_digests_max_digest_length (default 2048): Controls the maximum length of stored digest text. Reducing this caps per-entry memory at the cost of truncating long queries.
mysql-query_digests_grouping_limit (default 3): Controls how many consecutive query elements are preserved before grouping in the normalized digest. Lower values produce more aggressive grouping, reducing distinct digest count at the cost of coarser query identification. After changing variables:

LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

Query cache at capacity

Immediate relief:

PROXYSQL FLUSH QUERY CACHE;

This frees all cached result sets instantly. The cache will refill as queries arrive, causing temporary cache misses.

For sustained control:

Reduce cache_ttl on query rules caching high-cardinality queries. Queries with UUIDs, timestamps, or other unique literals in the cache key produce near-zero hit rates while consuming full cache entries.
Reduce mysql-query_cache_size_MB if the cache is consuming a disproportionate share of memory.
Set cache_empty_result=0 on query rules that frequently return empty result sets. This prevents wasting cache space on results that deliver no hit-rate value.
Identify which rules have caching enabled: SELECT rule_id, match_digest, cache_ttl FROM runtime_mysql_query_rules WHERE cache_ttl > 0;

Per-connection buffer pressure

Buffer memory scales with active connections. The levers are:

Reduce connection count: Fix multiplexing collapse. If Client_Connections_hostgroup_locked / Client_Connections_connected is high, investigate which session variables or prepared statements are pinning connections. See the ProxySQL client connections at mysql-max_connections guide for frontend saturation details.
Check mysql-stacksize (default 1048576 bytes, or 1 MB per thread): Thread stack memory is tracked as stack_memory_mysql_threads in stats_memory_metrics. With many worker threads (mysql-threads), stack memory alone can be significant.
Check mysql-threshold_resultset_size (default 4 MB): ProxySQL pauses backend result fetching when buffered-but-unsent data exceeds 8 times this value. Slow clients or large result sets can cause buffers to accumulate.

Prepared statement metadata

Prepared statements consume memory in the global cache and per-connection state. They also disable multiplexing for affected sessions, compounding buffer pressure.

mysql-max_stmts_cache (default 10000): Bounds the global prepared statement cache. Reduce if applications prepare many unique statements without deallocating.
mysql-max_stmts_per_connection (default 20): When this many statements are prepared on a single backend connection, ProxySQL closes and resets that connection. This bounds per-connection statement memory.

Check current counts:

SELECT Variable_Name, Variable_Value FROM stats_mysql_global
WHERE Variable_Name IN ('Stmt_Cached', 'Stmt_Client_Active_Total', 'Stmt_Server_Active_Total');

jemalloc fragmentation and retention

If fragmentation (jemalloc_active - jemalloc_allocated) is the problem, the workload pattern is likely creating many short-lived allocations of varying sizes. Options:

Restart ProxySQL during a maintenance window: The blunt instrument. A fresh process starts with a compact arena. Use only as a temporary measure while you address the root cause.
Disable retention: Start ProxySQL with MALLOC_CONF="retain:false" to prevent jemalloc from retaining freed memory. This reduces jemalloc_retained at the cost of potentially higher CPU on repeated allocation patterns.
Enable jemalloc profiling: ProxySQL supports jemalloc’s built-in profiling via MALLOC_CONF="prof:true". Dump profiles with jemalloc’s standard mechanisms (MALLCTL or signal-based dumping) to identify which code path is leaking. This is the definitive way to locate a genuine leak.

Version-specific leaks

If the growth pattern does not match any of the above causes and RSS climbs monotonically regardless of workload, you are likely hitting a known bug. Check the ProxySQL release notes for your version. Upgrade to the latest stable release in your tier.

Prevention

Set up periodic digest resets. Schedule SELECT * FROM stats_mysql_query_digest_reset at a regular interval (hourly or daily depending on workload cardinality). This prevents unbounded digest accumulation without requiring manual intervention during incidents.

Monitor the correlation, not just the absolute value. Alert on RSS growth that is uncorrelated with Client_Connections_connected growth, not on RSS alone. An absolute RSS threshold pages you during every traffic spike. A decorrelation alert fires only when something is wrong.

Cap the query cache deliberately. Set mysql-query_cache_size_MB based on available system memory, not the default. Verify that cached queries actually have meaningful hit rates. A cache at capacity with a low hit ratio is pure overhead.

Track VmSwap as a hard alert. Any swap usage by ProxySQL means performance has already degraded. This should be a page, not a ticket.

Keep ProxySQL current. Memory leak fixes land regularly. Running an old version with a known leak is a self-inflicted problem.

How Netdata helps

Per-second metric collection: Netdata collects jemalloc_resident, query_digest_memory, and the full stats_memory_metrics breakdown at 1-second resolution, making growth trends visible immediately rather than after multiple polling intervals.
Correlation of RSS with connection count: Netdata dashboards let you overlay jemalloc_resident with Client_Connections_connected in the same view, making the leak-versus-load determination visual rather than a manual delta exercise.
Anomaly detection: Netdata’s anomaly detector learns the normal growth pattern for your ProxySQL instance and flags deviations that a static threshold would miss.
VmSwap detection: Netdata surfaces swap usage at the process level, catching degradation before the OOM killer fires.