$ guides / proxysql / proxysql-too-many-open-files ▌

Operations Guides

ProxySQL Too many open files: file descriptor exhaustion takes down connections and health checks

ProxySQL hits a host-level cliff that its own stats tables cannot see. Every client, backend, monitor, and admin session consumes a file descriptor. When the process reaches its RLIMIT_NOFILE ceiling, accept() and connect() fail simultaneously: new client connections are rejected, backend connections cannot be established, and the monitor module cannot open sockets to check backend health. The proxy appears to hang while the process keeps running.

The first visible symptom is often not a connection error. ProxySQL’s internal SQLite database also needs file descriptors. When the pool is exhausted, SQLite fails first with unable to open database file, which operators frequently misdiagnose as disk corruption or a permissions problem. Connection errors surface after that: 2004: Can't create TCP/IP socket (24) in the ProxySQL log, where errno 24 is EMFILE.

ProxySQL does not expose file descriptor usage in any stats_mysql_* table. There is no counter for current FD consumption, no warning when connection limits exceed the FD budget, and no alerting before the cliff. Feature request #2595 asking for FD limit warnings remains open. Operators must collect FD usage from the OS and correlate it with ProxySQL’s connection counters manually.

FD budget breakdown

File descriptor exhaustion is binary: the proxy works until the limit is hit, then everything fails at once. The failure is invisible to ProxySQL’s own observability layer because the stats tables operators rely on (stats_mysql_global, stats_mysql_connection_pool) do not include an FD counter.

The FD budget must cover all simultaneous consumers:

flowchart TD
    FD["Process RLIMIT_NOFILE
e.g. 102400 file descriptors"] --> CL["Client connections
(1 FD each)"]
    FD --> BK["Backend connections
(1 FD each)"]
    FD --> MN["Monitor connections
(~2-3 FDs per backend)"]
    FD --> AD["Admin connections
(1 FD each)"]
    FD --> SQ["SQLite + internal FDs
(overhead)"]
    CL --> EXH["Exhaustion: accept fails
connect fails
monitor checks fail"]
    BK --> EXH
    MN --> EXH

When any consumer grows faster than expected (a connection storm after failover, a multiplexing collapse pinning backend connections 1:1, or an FD leak), the shared budget drains silently until the process hits the wall.

Common causes

Cause	What it looks like	First thing to check
FD limit too low for configured `mysql-max_connections`	Process has plenty of client capacity configured but `ulimit -n` is 1024 or 65536. Failure happens well before ProxySQL’s own limits.	`cat /proc/$(pidof proxysql)/limits \| grep "Max open files"`
Connection storm after failover or restart	Tens of thousands of client reconnects in seconds. FD count spikes to limit, then all new connects fail with errno 24.	ProxySQL log for `Client_Connections_created` rate; check if a failover or restart just occurred.
Multiplexing collapse pinning backend connections	Backend connection count tracks client connection count nearly 1:1. Each pinned session holds an FD that multiplexing would otherwise share.	`Client_Connections_hostgroup_locked / Client_Connections_connected` ratio.
FD leak (SSL CA certificate handling, versions 2.5.1-2.5.4)	FD count grows monotonically without corresponding connection growth. Process eventually crashes and restarts.	ProxySQL version; whether `mysql_servers.use_ssl` is enabled.
Effective systemd limit lower than expected	Process inherits 1024 FDs despite the unit file setting `LimitNOFILE=102400`. Can result from packaging differences or manual start inheriting the shell ulimit.	`systemctl show proxysql \| grep LimitNOFILE`

Quick checks

# Current FD limit for the ProxySQL process
cat /proc/$(pidof proxysql)/limits | grep "Max open files"

# Current FD count
ls -1 /proc/$(pidof proxysql)/fd | wc -l

# Effective systemd limit (not just the unit file)
systemctl show proxysql | grep LimitNOFILE

# ProxySQL connection-related settings
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM global_variables WHERE variable_name IN ('mysql-max_connections','mysql-threads','mysql-free_connections_pct');"

# Current client and server connection counts
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM stats_mysql_global WHERE variable_name IN ('Client_Connections_connected','Server_Connections_connected','Client_Connections_hostgroup_locked');"

# Sum of backend max_connections (potential FD ceiling if all pools fill)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT SUM(max_connections) AS total_backend_max FROM runtime_mysql_servers;"

# EMFILE errors in the ProxySQL log
grep -i "Can't create TCP/IP socket" /var/lib/proxysql/proxysql.log | tail -20

# SQLite FD exhaustion errors
grep -i "unable to open database file" /var/lib/proxysql/proxysql.log | tail -20

# ProxySQL version (relevant for the 2.5.1-2.5.4 SSL FD leak)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 -e "SELECT @@version;"

How to diagnose it

Confirm the FD limit and current usage. Compare /proc/<pid>/limits against the live FD count from /proc/<pid>/fd. If usage is at or near the limit, exhaustion is confirmed.
Check whether ProxySQL tried to auto-increase the limit. ProxySQL 2.0+ calls setrlimit(RLIMIT_NOFILE, ...) at startup to raise the soft limit to 102400 if it detects a low limit. Search the log for [WARNING] Automatically setting RLIMIT_NOFILE to 102400. If the hard limit is also low, the auto-increase fails silently and ProxySQL runs with the lower limit.
Distinguish a leak from a capacity problem. If FD count grows without corresponding growth in Client_Connections_connected or Server_Connections_connected, suspect a leak. If FD count tracks connection count proportionally, the limit is simply too low for the workload.
Identify which consumer is dominant. Sum the live connection counts: Client_Connections_connected + Server_Connections_connected + monitor connections (roughly backends x 3) + admin connections. Compare the total to the FD count. If the math accounts for most FDs, the issue is capacity. If there is a large unexplained gap, a leak is more likely.
Check the effective systemd limit. The official ProxySQL unit file sets LimitNOFILE=102400, but some distribution packages ship a different unit file without this directive. Run systemctl show proxysql | grep LimitNOFILE. If the process was started manually, it inherits the shell’s ulimit, which may be 1024.
Check the ProxySQL version if SSL is enabled. Versions 2.5.1 through 2.5.4 have a confirmed FD leak when using SSL CA certificates for backend connections. Fixed in 2.5.5.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`/proc/<pid>/fd` count vs. `RLIMIT_NOFILE`	Direct measure of FD headroom. ProxySQL stats tables do not expose this.	Usage above 50% of the limit.
`Client_Connections_connected`	Largest FD consumer. Each client session holds 1 FD.	Count approaching `mysql-max_connections` with FD usage tracking proportionally.
`Server_Connections_connected`	Backend connections also cost 1 FD each. Multiplexing collapse makes this grow linearly with clients.	Backend count approaching client count (ratio near 1:1).
`Client_Connections_hostgroup_locked`	Pinned sessions cannot share backend connections, inflating backend FD usage.	`hostgroup_locked / connected` ratio above 50%.
ProxySQL log: `Can't create TCP/IP socket (24)`	Definitive FD exhaustion error. Errno 24 is `EMFILE`.	Any occurrence means the limit was hit.
ProxySQL log: `unable to open database file`	SQLite fails when no FDs are available. Often the first visible symptom.	Appears before connection errors in some incidents.
`ProxySQL_Uptime`	Short uptime after a crash suggests the process hit the FD wall and was restarted.	Recent restart correlated with FD exhaustion errors in the previous boot’s logs.

Fixes

Raise the FD limit

The immediate fix is to raise RLIMIT_NOFILE so the process has headroom for peak connection counts plus overhead.

systemd-managed ProxySQL: Create a drop-in override:

systemctl edit proxysql

Add:

[Service]
LimitNOFILE=1048576

Reload and restart. Warning: systemctl restart proxysql drops all active client connections.

systemctl daemon-reload
systemctl restart proxysql

After restart, verify the effective limit:

cat /proc/$(pidof proxysql)/limits | grep "Max open files"

Non-systemd or manual start: Set the hard and soft limits before starting ProxySQL:

ulimit -n 1048576
proxysql -c /etc/proxysql.cnf

ProxySQL 2.0+ will attempt to auto-increase the soft limit to 102400 if it detects a low limit, but this only works if the hard limit permits it. If both hard and soft limits are 1024, the auto-increase fails silently.

Fix the FD leak (versions 2.5.1-2.5.4 with SSL)

If ProxySQL is running version 2.5.1 through 2.5.4 and backend connections use SSL (use_ssl=1 in mysql_servers), the x509 cache patch leaks a file descriptor on every new SSL connection. Upgrade to 2.5.5 or later.

Check whether SSL is enabled on backends:

SELECT hostgroup_id, hostname, port, use_ssl FROM runtime_mysql_servers WHERE use_ssl > 0;

Reduce FD pressure from connection storms

If the limit is adequate for steady-state but a connection storm temporarily exhausts FDs:

Stagger application restarts and use rolling deployments so not all clients reconnect at once.
Configure mysql-free_connections_pct (default 10%) to maintain a warm idle pool, reducing synchronous connection creation during bursts.
Size the FD limit for burst traffic, not just steady-state load. A practical target is at least 4x peak client connections.

Address multiplexing collapse

If backend connections are pinned nearly 1:1 to client sessions (visible as a high Client_Connections_hostgroup_locked count), the FD budget is consumed faster than necessary. See ProxySQL backend connection pool exhausted: queries queuing for a free connection for diagnosis steps. The short-term fix is to raise the FD limit. The long-term fix is to identify which session variables or transaction patterns are disabling multiplexing.

Prevention

Size the FD limit correctly. ulimit -n should be at least 4x peak expected client connections. A more precise calculation:

FD budget >= peak_client_connections
           + sum(mysql_servers.max_connections across all hostgroups)
           + (num_backends x 3)        # monitor connections
           + 1000                      # SQLite, admin, overhead

Alert when FD usage exceeds 50% of the limit. This catches a growing leak or a connection storm before exhaustion.

Do not rely on ProxySQL to warn you. ProxySQL does not validate whether configured mysql-max_connections fits within the FD budget. Settings can change at runtime without warning if the math does not work out.

Audit the effective FD limit after upgrades. Package upgrades can replace the unit file. Some distribution packages omit LimitNOFILE. Verify after every upgrade with systemctl show proxysql | grep LimitNOFILE.

Monitor FD count from the OS, not from ProxySQL. Collect /proc/<pid>/fd count alongside ProxySQL connection metrics. If FD count and Client_Connections_connected + Server_Connections_connected diverge (FDs growing without connection growth), investigate a leak.

How Netdata helps

Netdata collects OS-level file descriptor usage per process at per-second resolution via /proc/<pid>/fd, providing the signal ProxySQL’s stats tables cannot expose.
The ProxySQL collector surfaces Client_Connections_connected, Server_Connections_connected, and connection pool metrics from stats_mysql_global and stats_mysql_connection_pool, allowing correlation of connection growth against FD usage on the same timeline.
ML-based anomaly detection can flag unexpected FD growth patterns (such as a slow leak from an SSL library bug) before the process hits the cliff, even when absolute thresholds look normal.
Backend connection pool metrics (ConnUsed, ConnFree, ConnERR per backend) help identify whether multiplexing collapse or a specific backend is driving FD pressure.