PgBouncer query_wait_timeout: clients disconnected after waiting too long for a connection

Your PgBouncer log is filling with pooler error: query_wait_timeout lines and application teams are reporting intermittent database errors. The error string sounds like a query problem. It is not. query_wait_timeout fires when a client has been sitting in PgBouncer’s wait queue, blocked on getting a server connection, for longer than the configured timeout. The client never reached PostgreSQL.

The default is 120 seconds. When this error appears, the pool has been exhausted for at least that long. query_wait_timeout is a lagging indicator: it tells you an incident happened, not that one is starting.

A trap makes this worse in most deployments. If your application-side timeout (5 to 30 seconds is typical) is shorter than query_wait_timeout, the application gives up and retries long before PgBouncer ejects the waiter. Each retry adds a new client to the queue. PgBouncer’s timeout never saves you; it just cleans up abandoned waiters while the retry storm deepens the queue.

What this means

PgBouncer maintains one connection pool per (database, user) pair, sized by pool_size. When all server connections in a pool are busy, incoming clients enter a FIFO wait queue. The age of the oldest waiter is exposed as maxwait in SHOW POOLS. When a waiter’s age exceeds query_wait_timeout (default 120s), PgBouncer disconnects that client with an error and logs the event.

The disconnection is a cleanup mechanism, not a rescue. When the application timeout is shorter than query_wait_timeout, the app times out and retries first, and the PgBouncer timeout only reaps the abandoned queue entries. The retry amplification loop looks like this:

flowchart TD
  A[All server connections busy] --> B[Clients enter wait queue]
  B --> C[App timeout fires at 5-30s]
  C --> D[App retries with new connection]
  D --> B
  B --> E[Queue grows faster than it drains]
  E --> F[Waiter age hits query_wait_timeout at 120s]
  F --> G[PgBouncer disconnects waiter and logs error]
  G --> H[Log-only event: no SHOW counter exists]

Two things to note about the mechanics:

• maxwait uses the oldest queued client’s query_start; if that is zero, the janitor falls back to the client’s request_time. In normal query waiting, this measures from when the query was sent, not when the client connected. A client can be connected for hours in session mode; the clock starts on the pending query.
• The event is log-only. PgBouncer exposes no error counter for it in any SHOW command. If you only scrape SHOW output, you will not see these disconnections at all.

Do not confuse query_wait_timeout with query_timeout. The latter cancels queries that run too long on the backend (default 0, disabled). query_wait_timeout disconnects clients that never got a server connection. Setting query_timeout has no effect on wait-queue behavior.

Common causes

The paused/disabled check matters because PAUSE produces exactly the same signal shape as an outage: high cl_waiting, zero sv_active, waiters aging toward the timeout. Rule it out before treating this as an incident.

Quick checks

All of these are read-only and safe during an incident. They assume the admin console on port 6432; adjust for your deployment.

# 1. Admin console responsiveness - if this is slow, the event loop itself is impaired
time psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW VERSION;" > /dev/null

# 2. Pool state: which pool is saturated, how deep is the queue, how old is the oldest waiter
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW POOLS;"

# 3. Paused/disabled context before escalating anything
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW DATABASES;"

# 4. Averages: is the backend slow (query_time) or is the pool just small (wait_time)?
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW STATS_AVERAGES;"

# 5. Timeout events from the log (the only place they exist)
grep -cE "query_wait_timeout" /var/log/pgbouncer/pgbouncer.log
grep -E "timeout" /var/log/pgbouncer/pgbouncer.log | tail -20

# 6. Per-connection detail: which server connections are holding the pool
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW SERVERS;"

# 7. Per-waiter detail: how close each waiting client is to the timeout
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW CLIENTS;"

# 8. Verify the configured timeout actually is what you think it is
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW CONFIG;" | grep -i wait

In SHOW POOLS, the columns that matter are cl_waiting (queue depth per pool), sv_active (compare against pool_size), sv_idle (headroom), and maxwait plus maxwait_us (oldest waiter’s age). In SHOW CLIENTS, wait and wait_us show how long each waiting client has been blocked and how close it is to being ejected.

How to diagnose it

1. Confirm the pool is saturated, not just busy. In SHOW POOLS, find the pool where cl_waiting > 0. Check that sv_active equals pool_size and sv_idle is zero. If sv_active is well below pool_size while clients wait, something else is wrong: check for a paused database or backend connection failures.

2. Check paused/disabled state. SHOW DATABASES shows paused and disabled per database. A paused database queues all new queries while existing transactions complete. This is expected during maintenance and is not an incident.

3. Attribute the latency: pool or database. Compare avg_wait_time against avg_query_time in SHOW STATS_AVERAGES. High wait time with low query time means the database is fine and the pool is too small. High query time means PostgreSQL is slow and connections are held longer, which shrinks effective pool capacity.

4. Distinguish pool exhaustion from backend failure. In pool exhaustion, the total server connection count is stable at pool_size with everything in sv_active. In backend failure, the total is declining and sv_login is elevated as connections attempt and fail to establish. SHOW DNS_HOSTS tells you whether resolution is returning current addresses; a stale cache after failover points PgBouncer at a dead backend.

5. Find the connections holding the pool. SHOW SERVERS lists every server connection with state and request_time. An active connection with an old request_time is a long-running query or transaction monopolizing a slot. The link column lets you trace it back to the responsible client in SHOW CLIENTS.

6. Check for idle-in-transaction. If avg_xact_time is many times larger than avg_query_time, applications are holding transactions open while doing non-database work. Confirm on the PostgreSQL side with pg_stat_activity filtered on idle in transaction.

7. Reconstruct the timeline from the log. Count and timestamp the query_wait_timeout lines. Subtract your configured timeout from the first occurrence to get when the saturation event started.

Metrics and signals to monitor

Alert on sustained maxwait above a threshold, not on cl_waiting > 0. Brief queuing during bursts is normal in transaction pooling mode and will drown you in false positives. Set the threshold relative to your application timeout, not an absolute number.

Fixes

Fix the immediate saturation

Identify long-running queries or transactions holding server connections via SHOW SERVERS (old request_time on active connections), then cancel them on the PostgreSQL side with pg_cancel_backend or pg_terminate_backend. Coordinate with the application team first: terminating a backend errors the client attached to it through PgBouncer, so this is a disruptive action.

Do not restart PgBouncer as a first response. A restart drops all cached server connections and every client triggers re-authentication against PostgreSQL simultaneously, producing a thundering herd that is often worse than the original saturation.

Right-size the pool

If the database is healthy and avg_query_time is at baseline, the pool is too small. Increase pool_size (per-database override or default_pool_size) and issue RELOAD. Constraint: the sum of all PgBouncer pools’ pool_size values must fit within PostgreSQL’s max_connections with room for superuser connections, direct access, and replication. Multiple PgBouncer instances targeting the same backend multiply the demand.

If reserve_pool_size is configured and you see reserve connections drawn regularly (total server connections per pool exceeding pool_size, plus taking connection from reserve_pool warnings in the log), the base pool is chronically undersized. The reserve is for brief spikes; sustained usage masks the real sizing problem.

Fix the application behavior

If avg_xact_time >> avg_query_time, the fix is in the application: transactions held open across non-database work. On the PostgreSQL side, idle_in_transaction_session_timeout can auto-terminate these as a guardrail while the code is fixed.

Align the timeouts

This is the fix specific to this error. The relationship that matters:

• Application timeout shorter than query_wait_timeout (the common case: 5-30s vs 120s) produces retry amplification. The app retries, the queue deepens, and the PgBouncer timeout only reaps corpses.
• query_wait_timeout = 0 disables the timeout entirely, so clients queue indefinitely. That trades disconnection errors for unbounded queue growth and is rarely what you want.

Either lower query_wait_timeout to a value near the application’s retry budget so PgBouncer fails waiters fast and predictably, or keep it high only if clients genuinely cannot retry and must wait. What you cannot do is leave a 120-second PgBouncer timeout under a 10-second application timeout and expect the log lines to mean anything other than “we have been saturated for a very long time.”

Prevention

• Alert on the leading indicators, not the timeout. Sustained maxwait and sv_active / pool_size above 85% fire minutes before the first disconnection. The query_wait_timeout log line is the post-mortem.
• Parse the log. PgBouncer exposes no error counters via SHOW commands. Timeout events, connection refusals, and auth failures exist only in the log file. A monitoring setup that ignores logs is blind to every error condition PgBouncer reports.
• Track timeout alignment in configuration review. Whenever application timeout settings change, check them against query_wait_timeout. Drift here is common and invisible until an incident.
• Set max_client_conn with the retry storm in mind. During a queueing event, retries create new client connections. If used_clients reaches max_client_conn, new connections are refused with no more connections allowed, converting a latency incident into a hard outage.
• Load-test the queue behavior. The interaction between pool size, application timeout, retry policy, and query_wait_timeout is only observable under saturation. A controlled test that drives the pool to exhaustion will show you which timeout fires first and how the retry cascade behaves.

How Netdata helps

• Per-second cl_waiting and maxwait collection catches sub-minute queueing events that interval polling misses, and timestamps the start of saturation precisely instead of inferring it from the first log line 120 seconds later.
• Wait time vs query time correlation puts avg_wait_time and avg_query_time on the same timeline, so “pool too small” and “database slow” are distinguishable at a glance rather than requiring two separate queries during an incident.
• Pool utilization trending (sv_active against pool_size, sv_idle headroom) shows the slow creep toward saturation days before the first query_wait_timeout appears, turning a reactive fix into a capacity change.
• Anomaly detection on maxwait flags the transition from normal burst queuing to sustained starvation without a hand-tuned static threshold.
• Cross-pool visibility surfaces the single saturated (database, user) pool that aggregate dashboards hide behind healthy averages.
• Log-based alerting on timeout events closes the gap left by PgBouncer’s lack of error counters, so disconnections page the right people instead of being discovered in a log grep the next morning.

Related guides

• How PgBouncer actually works in production: a mental model for operators
• PgBouncer monitoring checklist: the signals every connection pooler needs
• PgBouncer monitoring maturity model: from survival to expert
• PgBouncer pool exhaustion: clients queue, wait times climb, and the retry cascade