PgBouncer client connection leak: idle clients that never disconnect

used_clients keeps climbing. Traffic is flat, cl_waiting is zero, queries are fast, and yet the client count creeps toward max_client_conn day after day. When it gets there, PgBouncer starts rejecting new connections with no more connections allowed (max_client_conn), even though the database itself is completely healthy.

This is a client-side connection leak: application instances open connections to PgBouncer and never close them. PgBouncer is doing exactly what it is told to do, holding those sockets open. The pooler just makes the leak visible earlier and more painfully, because it has a hard front-door limit.

PgBouncer exposes enough state to find the leaking clients precisely, and recent versions (1.24+) give you tools to reap them without a restart. This guide covers detection, distinguishing a real leak from expected long-lived connections, and remediation.

What this means

Every client connected to PgBouncer consumes a file descriptor, a small amount of memory, and one slot out of max_client_conn. A client that connects and never disconnects holds all of that indefinitely, whether it ever sends a query or not.

The cost depends on what the leaked client is holding:

• Idle client, no transaction open (cheap). In transaction pooling mode, an idle client holds no server connection. It costs an FD, some memory, and a client slot. Thousands of these slowly march you toward max_client_conn, but they do not starve the server pool.
• Client that ran BEGIN and wandered off (expensive). In transaction mode, a client inside an open transaction holds a server connection in sv_active state for as long as the transaction stays open. Enough of these and the pool exhausts: cl_waiting grows, maxwait climbs, and you have the full pool exhaustion cascade on top of the leak. idle_transaction_timeout exists for exactly this case.

The failure ends in one of two places: the front door closes (used_clients hits max_client_conn and new connections are refused with no queuing and no warning), or the server pool starves (leaked in-transaction clients pin every server connection). This article is about the leak underneath both.

flowchart TD
  A[used_clients drifting up] --> B{SHOW CLIENTS: old connect_time, no recent request_time?}
  B -- no --> C[Not a leak: legitimate traffic or capacity issue]
  B -- yes --> D{Client holds a server link?}
  D -- yes --> E[Expensive: idle-in-transaction, pins sv_active]
  D -- no --> F[Cheap: idle client, holds FD + client slot only]
  E --> G[idle_transaction_timeout + app fix]
  F --> H[client_idle_timeout + app fix]

Common causes

The hardest judgment call is the third row: application connection pools are supposed to hold long-lived connections. An old connect_time alone is not a leak. The leak signature is growth uncorrelated with traffic, plus connections whose request_time never advances, plus connections from instances that no longer exist.

Quick checks

All read-only, run against the admin console. Adjust host and port for your deployment.

# Client capacity: used vs free slots
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW LISTS;" | grep -E "used_clients|free_clients"

# The configured limit
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW CONFIG;" | grep max_client_conn

# Full client table for offline analysis (state, addr, database, user, connect_time, request_time, ...)
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW CLIENTS;" > /tmp/clients.txt

# Connections per source address: which host or subnet is hoarding
awk -F'|' '{print $4}' /tmp/clients.txt | sort | uniq -c | sort -rn | head -20

# Are any old clients holding server connections? (expensive leak)
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW SERVERS;" > /tmp/servers.txt

# Transaction hold time vs query time: a large gap means idle-in-transaction
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW STATS_AVERAGES;"

# FD pressure on the PgBouncer process
ls /proc/$(pgrep -f pgbouncer)/fd | wc -l
grep "Max open files" /proc/$(pgrep -f pgbouncer)/limits

Column positions in the -At output are version-dependent; the SHOW CLIENTS field order changed when columns were added in 1.23, so verify which field is addr on your version before trusting the awk position. Safer: run SHOW CLIENTS; once in aligned mode to see the header, then fix the field index.

How to diagnose it

1. Confirm the growth pattern. Sample used_clients from SHOW LISTS a few times over an hour. A leak grows monotonically or in steps (at deploy time) and never comes down. Legitimate load tracks traffic and falls during quiet periods. Also check query_count in SHOW STATS_AVERAGES: flat or falling query rate with rising client count is the smoking gun.

2. Identify the aged clients. From the captured SHOW CLIENTS output, find rows where connect_time is hours or days old and request_time is equally stale (no recent activity). On PgBouncer 1.25.0+, SHOW CLIENTS has an explicit idle state, which makes this trivial. On earlier versions there is no idle state; idle clients appear as active or used and you must infer idleness by comparing connect_time against request_time.

3. Split cheap leaks from expensive leaks. For each suspect client, check the link column. A non-empty link means the client is paired with a server connection. Cross-reference SHOW SERVERS: an active server connection with an old request_time whose state in PostgreSQL’s pg_stat_activity is idle in transaction is the expensive variant. These clients are eating pool capacity right now, not just client slots.

4. Attribute by source. Group the leaked connections by addr and by database/user. One source IP with thousands of connections points at a single misbehaving instance or a leaked pool in one service. Connections from IPs that no longer map to any live instance confirm orphaned connections from retired deployments.

5. Rule out look-alikes. Check SHOW DATABASES for paused or disabled flags before treating anything as an incident. Confirm that the count is not simply (app pool size x instance count) doing exactly what the app was configured to do; if instances scale out and each holds 20 connections, used_clients climbs legitimately and the fix is capacity, not leak hunting.

6. Quantify headroom. Compare used_clients against max_client_conn, and the process FD count against the ulimit. Above 80% of either, you are one deploy or one traffic spike away from refusals. Admin connections to the pgbouncer database are exempt from max_client_conn, so you will not lock yourself out of the console.

Metrics and signals to monitor

Note the instrumentation gap: PgBouncer has no counter for refused connections. no more connections allowed appears only in the log. If you are not parsing the log, the first signal of a full front door is application-side errors.

Fixes

Reap idle clients: client_idle_timeout

PgBouncer 1.24.0 added client_idle_timeout: client connections idle longer than N seconds are closed. Default is 0 (disabled). This is the direct fix for the cheap leak variant. Two cautions:

• The official docs flag this as a potentially dangerous timeout. Set it higher than the client-side pool’s own idle timeout and connection lifetime, otherwise PgBouncer will close connections the application still considers valid, and the app will see unexpected errors on its next query.
• It does nothing for the expensive variant. A client inside an open transaction is not idle in this sense.

Kill specific leaked clients: KILL_CLIENT

Also added in 1.24.0, KILL_CLIENT terminates a single client connection from the admin console. This is the surgical option for a handful of obvious orphans (for example, connections from an instance that was decommissioned last week) without touching healthy traffic. Prefer it over KILL, which drops every connection for a database.

Reap idle transactions: idle_transaction_timeout

For the expensive variant, idle_transaction_timeout (default 0, disabled) closes clients that sit inside an open transaction without doing work. This frees the pinned server connection back to the pool. Same caution applies: applications with legitimately long interactive transactions will get disconnected.

Cap per-user and per-database client counts

1.24.0 added max_user_client_connections and max_db_client_connections. These contain the blast radius: one leaking service exhausts its own allowance instead of the global max_client_conn and taking the whole front door down. Sizing these requires knowing your per-service connection budgets; it is a capacity planning exercise, not just a leak fix. See PgBouncer capacity planning.

Fix the application (the actual root cause)

Every timeout above is a leash, not a cure. The durable fixes are in the application: ensure connections are returned or closed on every code path, configure the app pool’s maximum lifetime and idle timeout to be shorter than PgBouncer’s reaping timeouts, and make sure deployments drain or kill old instances cleanly. If the app pool deliberately holds long-lived connections, that is fine, but then max_client_conn must be sized for (pool size x instance count) with headroom, and tuned against real FD limits as covered in PgBouncer max_client_conn tuning.

Restart as last resort, with a version caveat

A restart clears everything, but it is the blunt option: it drops all healthy connections too and triggers a reconnection storm against PostgreSQL. If you do restart, note that since 1.23.0, SIGTERM performs a “super safe shutdown” that waits for all clients to disconnect. With leaked clients that will never disconnect voluntarily, a SIGTERM shutdown can hang indefinitely. SIGQUIT gives the old immediate-shutdown behavior. Plan for that before you are staring at a hung shutdown during an incident.

Prevention

• Monitor the trend, not the threshold. Alert on used_clients / max_client_conn sustained above 80%, and separately on week-over-week growth of the ratio. Leaks are slow; a trend alert fires days before the wall.
• Baseline client age. Periodically snapshot SHOW CLIENTS and track the age distribution. Alert when connections are older than the oldest live application instance, since nothing legitimate outlives its own process.
• Set app-side lifetimes shorter than PgBouncer-side timeouts. The application should always recycle its own connections before the pooler kills them. That ordering keeps errors out of the app.
• Contain with per-user or per-database caps so one leaking service cannot consume the global limit (1.24+).
• Audit at deploy time. The step-up pattern (count rises at each deploy, never falls) means old instances are not draining. Fix graceful shutdown in the deploy pipeline.
• Parse the log. no more connections allowed, client_idle_timeout events, and query_wait_timeout events exist only in the log. Refusals and reaping events are your confirmation that a leak reached production impact.

How Netdata helps

Netdata’s PgBouncer collector queries the admin console continuously, so the signals this guide uses are already time-series rather than manual snapshots:

• Client slot utilization over time. used_clients and free_clients as a trend make the slow drift visible weeks before max_client_conn is hit, which is when a leak is still cheap to fix.
• Wait time vs query time correlation. avg_wait_time against avg_query_time tells you whether latency comes from the pool or the database; avg_xact_time against avg_query_time exposes the idle-in-transaction variant of the leak.
• Pool state correlation. If leaked transactions start pinning server connections, sv_active climbing toward pool_size alongside growing client counts shows the leak escalating from cheap to expensive.
• Anomaly detection on client counts. A step-change in client connections with flat query throughput is exactly the pattern ML-based anomaly flags catch, without you hand-tuning a growth threshold.
• Per-pool breakdown. Leaks are usually one service. Per-database and per-user views isolate the offender instead of averaging it away.

Related guides

• PgBouncer no more connections allowed (max_client_conn): the front door is full
• PgBouncer max_client_conn tuning: setting the client limit against real FD headroom
• PgBouncer pool exhaustion: clients queue, wait times climb, and the retry cascade
• PgBouncer pool utilization high: sv_active approaching pool_size before clients queue
• PgBouncer maxwait high: the oldest client waiter and how close it is to timing out
• PgBouncer avg_wait_time high: the latency the pool itself is injecting
• PgBouncer capacity planning: runway for pools, clients, and PostgreSQL slots
• PgBouncer monitoring checklist: the signals every connection pooler needs
• How PgBouncer actually works in production: a mental model for operators