PgBouncer pool exhaustion: clients queue, wait times climb, and the retry cascade

Every server connection in a pool is busy. sv_active equals pool_size, sv_idle is zero, and cl_waiting is climbing. Clients that were getting sub-millisecond connection assignment are now sitting in a FIFO queue, and the oldest waiter (maxwait) is old enough that application timeouts are firing. This is PgBouncer pool exhaustion, the most common PgBouncer incident, and it has a nasty property: it feeds itself.

The cascade works like this. Server connections are held too long, so new client requests queue. Application-side timeouts are almost always shorter than PgBouncer’s query_wait_timeout (default 120 seconds), so the application gives up first, retries, and adds a fresh waiter to the queue behind the one that never left. Retries arrive faster than the queue drains. cl_waiting snowballs until max_client_conn (default 100) is reached, at which point new connections are refused outright with "no more connections allowed (max_client_conn)" in the log. What started as one slow query becomes a full outage.

The failure is legible if you look at the right signals, and the first commands take seconds to run.

What this means

PgBouncer multiplexes many client connections onto a small pool of server connections, one pool per (database, user) pair. When a client sends a query and no server connection is free, the client enters a FIFO wait queue. The queue itself is normal during bursts; the pool is designed to absorb short spikes. The incident begins when the queue stops draining.

The degradation curve is cliff-edge, not gradual. Below 100% pool utilization, wait time is effectively zero. At 100%, latency jumps from zero to unbounded. There is no graceful middle state, which is why the transition from “fine” to “clients are timing out” feels instantaneous.

flowchart TD
    A[Slow queries or long transactions] --> B[Server connections held longer]
    B --> C[sv_active = pool_size, sv_idle = 0]
    C --> D[New clients enter FIFO queue: cl_waiting grows]
    D --> E[maxwait climbs past app timeout]
    E --> F[App times out and retries]
    F --> D
    D --> G[used_clients hits max_client_conn]
    G --> H[New connections refused: full outage]

The loop between “app times out and retries” and “cl_waiting grows” is the cascade. Breaking that loop, or draining the queue faster than retries refill it, is the whole game.

Common causes

The last two rows are not pool exhaustion, but they produce the same cl_waiting symptom. Rule them out before treating this as a capacity problem.

Quick checks

All commands run against the PgBouncer admin console. All are read-only.

# 1. Pool state: the core snapshot. Look for sv_active = pool_size,
#    sv_idle = 0, cl_waiting > 0, maxwait climbing. Also check sv_login:
#    rising sv_login with declining total server connections means
#    connections are failing to establish, not being held.
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW POOLS;"

# 2. Rule out maintenance. paused=1 or disabled=1 means cl_waiting
#    is expected behavior, not an incident.
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW DATABASES;"

# 3. Latency attribution: is the database slow or is the pool small?
#    Compare query_time (backend execution) against wait_time (queuing).
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW STATS_AVERAGES;"

# 4. Which specific server connections are stuck? Oldest request_time
#    in active state is the primary suspect.
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW SERVERS;"

# 5. Who is waiting and for how long? wait / wait_us per client, and
#    addr tells you which application instance is piling in.
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW CLIENTS;"

# 6. How close to the hard client limit? If the retry storm is mature,
#    used_clients approaches max_client_conn.
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW LISTS;"

# 7. Log evidence: max_client_conn refusals, query_wait_timeout
#    ejections, and backend "connect failed" messages. Log path comes
#    from the logfile setting in pgbouncer.ini (or journalctl -u pgbouncer).
grep -E "max_client_conn|query_wait_timeout|connect failed" /path/to/pgbouncer.log | tail -50

On the PostgreSQL side, one query finds the connections that are holding pool slots hostage:

# Find idle-in-transaction sessions holding PgBouncer server connections
psql -h <postgres-host> -U postgres -c \
  "SELECT pid, state, now() - xact_start AS xact_duration, query
   FROM pg_stat_activity
   WHERE state = 'idle in transaction'
   ORDER BY xact_duration DESC;"

How to diagnose it

1. Confirm the pattern. In SHOW POOLS, find the pool where sv_active equals the configured pool_size (cross-reference SHOW DATABASES for per-database pool sizes) and sv_idle is 0. If cl_waiting is 0, you have zero headroom but no incident yet. If cl_waiting is growing and maxwait is past 5 seconds, you are in the cascade.

2. Rule out the mimics. Check SHOW DATABASES for paused = 1 or disabled = 1; during a PAUSE, cl_waiting spikes and sv_active drains to zero by design. Then check whether total server connections (sv_active + sv_idle + sv_used + sv_tested + sv_login) are stable at pool_size or declining. Stable at pool_size with everything sv_active is exhaustion. Declining with sv_login elevated is backend connection failure, a different incident with a different fix.

3. Attribute the latency. Compare avg_wait_time against avg_query_time from SHOW STATS_AVERAGES (both in microseconds). High wait with normal query time means the pool is too small for the offered load. Elevated query time means PostgreSQL is the root cause and PgBouncer is the messenger. Then compare avg_xact_time to avg_query_time: if transaction time dwarfs query time, clients are holding server connections while doing no database work, the idle-in-transaction pattern.

4. Identify the offenders. In SHOW SERVERS, find active connections with the oldest request_time; those are the queries or transactions blocking pool turnover. Follow the link column to SHOW CLIENTS to get the source address of the responsible application. In SHOW CLIENTS, sort by wait / wait_us to see the longest waiters and how close they are to query_wait_timeout.

5. Assess cascade maturity. Check used_clients from SHOW LISTS against max_client_conn. If the gap is shrinking fast, retries are outpacing the drain rate and you have minutes before hard refusals begin. The log grep from quick check 7 confirms maturity: any query_wait_timeout event means a client waited the full timeout (default 120s) and was disconnected, and max_client_conn refusals mean the cascade has reached the wall.

Metrics and signals to monitor

Note on avg_wait_time: the calculation bug fixed in PgBouncer 1.23.0 divided accumulated wait time by the stats period rather than by the number of server assignments, so one long waiter could dominate the reported average. If you are on 1.22 or earlier, lean on maxwait and cl_waiting instead.

Fixes

Drain the queue: cancel the blocking queries

If SHOW SERVERS shows a small number of ancient active connections holding the pool, cancel them on the PostgreSQL side with pg_cancel_backend(pid), or pg_terminate_backend(pid) for sessions stuck idle in transaction. This is disruptive: the affected client gets an error, and pg_terminate_backend kills the session outright. Coordinate with the owning application team first, and prefer cancel over terminate. Freeing even a few slots lets the FIFO queue start draining, which often breaks the retry loop on its own.

Stop the bleeding: idle-in-transaction

If the root cause is transactions held open while the application does other work, the durable fix is application-side (commit promptly, do not span transactions across HTTP calls). As a backstop, set PostgreSQL’s idle_in_transaction_session_timeout so the backend auto-terminates abandoned transactions. This converts silent pool starvation into visible application errors, which is the right trade.

Right-size the pool

If avg_query_time is healthy and the database has capacity, the pool is simply undersized. A useful sizing relationship from the throughput side: required pool_size is roughly transactions_per_second x avg_xact_time_in_seconds. If that exceeds your configured pool_size, queuing is guaranteed at peak. Increase pool_size (or default_pool_size) and apply with RELOAD; no restart needed. Two constraints: the sum of all pool sizes across every PgBouncer instance targeting one PostgreSQL must stay under its max_connections with room for superuser and direct connections, and do not shrink pools because sv_idle looks high. In transaction mode, idle connections are the burst reserve, not waste.

Enable the reserve pool as a safety valve

With reserve_pool_size > 0 (default 0, disabled), PgBouncer opens extra server connections once a client has waited longer than reserve_pool_timeout (default 5s). This absorbs short spikes without permanently oversizing the base pool. Caveat: sustained reserve usage means the base pool is chronically undersized; watch for the "taking connection from reserve_pool" log warning and treat it as a sizing signal, not a solution. Reserve connections still consume PostgreSQL max_connections slots.

Align the timeouts

The retry cascade exists because application timeouts are shorter than query_wait_timeout. The client gives up, retries, and double-parks in the queue. You cannot make the queue drain faster than the underlying queries run, so decide deliberately: either lower query_wait_timeout so PgBouncer ejects hopeless waiters fast (failing fast beats queueing for two minutes), or raise it and fix application retry logic with backoff and jitter so retries stop compounding. Fixed-interval retries with no jitter are what turn linear overload into runaway queue growth.

If the event loop is the bottleneck

PgBouncer is single-threaded. If the admin console itself is slow to respond and per-process CPU is pinned at one core, no pool tuning will help; scale out with multiple processes via so_reuseport. This is rare but changes the fix entirely.

Prevention

• Alert on maxwait, not cl_waiting alone. Brief queuing during bursts is normal in transaction mode. Alert on maxwait > 15s sustained with cl_waiting > 0 and the database not paused. This filters bursts and maintenance windows. PgBouncer metrics cannot distinguish “overwhelmed by legitimate batch work” from “broken,” so these alerts are ticket-grade, not page-grade, unless paired with application-side error rates.
• Gate every PgBouncer alert on paused/disabled state. High cl_waiting with zero sv_active during a PAUSE is expected. Without this check you will get paged during every planned maintenance.
• Track the headroom trend. Watch sv_active / pool_size over weeks. Rising utilization with occasional short cl_waiting appearances is your runway signal; act at 85% sustained, before the cliff.
• Audit applications for idle-in-transaction. The avg_xact_time / avg_query_time ratio catches this before it causes an incident.
• Validate the connection budget. Sum of all PgBouncer pool sizes (plus reserve pools, plus all instances) must fit under PostgreSQL max_connections with margin, and max_client_conn must fit under the OS file descriptor limit minus server connections and overhead.
• Fix retry behavior application-side. Exponential backoff with jitter, and a retry budget. Retries that arrive faster than the queue drains are the cascade.

How Netdata helps

• Netdata collects PgBouncer pool metrics per (database, user) pool, so cl_waiting, sv_active, sv_idle, and maxwait are visible per pool rather than hidden inside a global aggregate. One saturated pool no longer masquerades as a healthy average.
• The avg_wait_time versus avg_query_time split is charted side by side, which makes the root-cause attribution step (“pool too small” versus “database slow”) a glance instead of two manual queries during an incident.
• Because Netdata samples at high frequency, it catches the short saturation spikes that a slower polling loop misses and that precede full cascades.
• Correlating PgBouncer signals with host-level PostgreSQL metrics on the same timeline lets you see whether elevated avg_query_time coincides with database CPU, I/O, or lock pressure, closing the loop from pool symptom to backend cause.
• Historical retention of sv_active / pool_size and maxwait gives you the trend data needed for the runway estimation in the prevention section.

Related guides

• PgBouncer maxwait high: the oldest client waiter and how close it is to timing out
• PgBouncer avg_wait_time high: the latency the pool itself is injecting
• PgBouncer query_wait_timeout: clients disconnected after waiting too long for a connection
• PgBouncer pool utilization high: sv_active approaching pool_size before clients queue
• PgBouncer sv_idle at zero: no headroom and one slow query from a cascade
• PgBouncer monitoring checklist: the signals every connection pooler needs
• How PgBouncer actually works in production: a mental model for operators
• PgBouncer monitoring maturity model: from survival to expert
• PgBouncer pool_size sizing: matching pool capacity to transaction time and throughput
• PgBouncer reserve pool activation: overflow capacity that hides an undersized pool
• PgBouncer capacity planning: runway for pools, clients, and PostgreSQL slots
• PgBouncer prepared statement does not exist: transaction pooling and lost session state