PgBouncer backend unreachable: PostgreSQL down and the pool draining

Clients are queueing in PgBouncer, wait times are climbing, and SHOW POOLS shows fewer server connections than there were an hour ago. Nobody changed the config. The likely cause: PostgreSQL is down, network-partitioned, or rejecting connections, and PgBouncer cannot establish new server connections to replace the ones it is losing.

This failure mode is deceptive because it degrades slowly. Existing server connections keep serving queries until they expire (server_lifetime, default 3600s), go idle past server_idle_timeout (default 600s), or error out. The pool drains gradually rather than failing all at once. Meanwhile clients pile into the wait queue and eventually get disconnected at query_wait_timeout (default 120s).

The other trap is that the failure is invisible in every SHOW counter. PgBouncer has zero error counters: no connect-failure count, no login-failure count. The root cause lives only in the log, in lines containing connect failed, S: login failed, or server DNS lookup failed. The metrics only show the consequences.

What this means

PgBouncer holds one pool per (database, user) pair, with server connections in states sv_active, sv_idle, sv_used, sv_tested, and sv_login. When the backend becomes unreachable:

1. New connection attempts enter sv_login and fail or stall.
2. Existing connections continue working, then close one by one as they expire or error.
3. The total server connection count (sv_active + sv_idle + sv_used) declines over time.
4. Clients queue in cl_waiting, maxwait grows, and the oldest waiters are disconnected at query_wait_timeout.

The distinguishing signature versus ordinary pool exhaustion is the total connection count trend, not the queue. In pool exhaustion the total is pinned at pool_size with everything in sv_active. In backend failure the total shrinks over time while sv_login stays elevated or fluctuating.

flowchart TD
  A[PostgreSQL down or unreachable] --> B[New server connections fail in sv_login]
  A --> C[Existing connections keep serving queries]
  C --> D[Connections expire via server_lifetime / server_idle_timeout or error]
  D --> E[Total pool size declines over time]
  B --> E
  E --> F[Clients queue: cl_waiting grows, maxwait climbs]
  F --> G[Oldest waiters disconnected at query_wait_timeout]
  H[Log: connect failed / S: login failed] -.only evidence of root cause.-> B

Common causes

Quick checks

All of these are read-only. Run them from the PgBouncer host.

# 1. Pool state: is the total server connection count declining?
psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -Atc "SHOW POOLS;"
# Watch sv_active + sv_idle + sv_used per pool. Falling total = draining pool.
# Stable total at pool_size with sv_idle=0 = exhaustion, a different problem.

# 2. Are connections stuck trying to log in?
psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -Atc "SHOW POOLS;" | awk -F'|' '{print $1, $2, "sv_login="$13}'
# Sustained sv_login > 0 with low sv_active means connections are failing to establish.

# 3. Rule out administrative state before escalating
psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -Atc "SHOW DATABASES;"
# Check paused and disabled columns. A PAUSE produces the same queue with zero sv_active.

# 4. DNS state for hostname-configured backends
psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -Atc "SHOW DNS_HOSTS;"
# Empty addrs or an obviously stale IP = the backend "outage" is a DNS problem.

# 5. The log: where the actual root cause lives
tail -200 /var/log/pgbouncer/pgbouncer.log | grep -iE "connect failed|login failed|DNS lookup failed"
# Adjust the path to your logfile setting.

# 6. Does PostgreSQL accept a direct connection, bypassing PgBouncer entirely?
psql -h <postgres-host> -p 5432 -U <user> -d <db> -c "SELECT 1;"

# 7. Is PostgreSQL refusing because it is full?
# On PostgreSQL itself:
psql -c "SELECT count(*), current_setting('max_connections')::int FROM pg_stat_activity;"

How to diagnose it

1. Confirm the symptom class. Pull SHOW POOLS twice, a minute apart. If sv_active + sv_idle + sv_used per pool is declining between samples, this is backend connection failure. If the total is stable at pool_size and everything is sv_active, you want the pool exhaustion guide instead (linked below).

2. Read the log first. This inverts the usual habit of starting with metrics, but here the metrics only show consequences. grep the log for connect failed, S: login failed, and server DNS lookup failed. The S: prefix marks a server-side (backend) event. The message after S: login failed: is the FATAL error PostgreSQL returned, which usually names the cause directly: recovery mode, too many connections, auth failure.

3. Test the backend directly, from the PgBouncer host. A direct psql connection that succeeds tells you PostgreSQL is alive and the problem is PgBouncer-side (credentials, config, DNS cache). A direct connection that fails tells you the problem is below PgBouncer: PostgreSQL down, network path broken, or PostgreSQL full.

4. If direct connection succeeds but logins still fail, check PostgreSQL’s connection count. If pg_stat_activity is at max_connections, PgBouncer cannot open new server connections even though the backend is healthy. Existing PgBouncer connections keep working; new ones get S: login failed. This looks like a backend outage in the queue but is actually a capacity collision.

5. If the backend is configured by hostname, check DNS. SHOW DNS_HOSTS shows the cached address and TTL. After a failover, PgBouncer may hold the old primary’s IP until the cache TTL expires. A RELOAD flushes the DNS cache; the RECONNECT command closes all server connections and forces new ones (available since 1.11). RECONNECT drops all existing healthy connections too, so use it deliberately, not reflexively.

6. Check the failure scope. All pools failing at once points to PostgreSQL, the network, or DNS. One pool failing while others are healthy points to per-database credentials, a per-database connection string, or a single backend behind a multi-backend config.

One behavioral detail that surprises operators: after a failed login, PgBouncer waits server_login_retry (default 15s) before trying again, and during that interval new clients that need a server connection get an immediate error instead of queueing. So the user-facing symptom can flip between “requests hang then time out” and “requests fail instantly” on a 15-second cycle.

Metrics and signals to monitor

Fixes

PostgreSQL is actually down

Fix PostgreSQL. PgBouncer needs no intervention: once the backend accepts connections, failed logins stop, the retry succeeds within server_login_retry, and the pool refills on demand. Expect sv_login to spike and cl_waiting to drain over the following seconds as queued clients get served. If the queue is deep, the drain takes a few pool turnovers, not zero time.

PostgreSQL is up but at max_connections

PgBouncer’s total possible server connections (sum of all pools’ pool_size plus reserve_pool_size, across every PgBouncer instance targeting this backend) must fit inside max_connections with room for superuser and direct access. The immediate lever is reducing demand or raising max_connections on PostgreSQL (requires a PostgreSQL restart). The durable fix is capacity planning so the arithmetic holds under failover and burst conditions.

Credentials rotated without updating PgBouncer

Update the credentials PgBouncer uses to authenticate to PostgreSQL, then RELOAD. If clients authenticate via auth_query, verify the auth function and auth_user permissions still work, since a broken auth_query also surfaces as login failures.

DNS is stale or failing

If SHOW DNS_HOSTS shows the old IP after a failover, RELOAD flushes the cache. Consider whether the DNS cache TTL is too long for your failover mechanism; the tradeoff of lowering it is more resolver load. If resolution itself is failing (server DNS lookup failed), fix the resolver path. Backends configured by IP bypass this class of failure entirely.

The queue is already deep

Do not restart PgBouncer. A restart discards every server connection, including the healthy ones still serving queries, and triggers a login storm against a backend that may already be struggling. Let query_wait_timeout (120s default) shed the stalest waiters; clients with their own shorter timeouts will already have given up and possibly retried, which is worth remembering when you size the retry wave after recovery.

Prevention

• Log capture is non-negotiable. Connect failures, login failures, and DNS failures appear nowhere in SHOW output. If your monitoring only scrapes SHOW commands, this failure class is invisible until clients start timing out. Ship the PgBouncer log and alert on connect failed and S: login failed.
• Alert on the draining signature, not just the queue. Track total server connections per pool as a time series. A declining total with elevated sv_login catches backend failure minutes before maxwait approaches query_wait_timeout.
• Gate every queue alert on paused/disabled. A PAUSE during maintenance produces identical cl_waiting and maxwait symptoms. Check SHOW DATABASES state in the alert condition.
• Keep the connection arithmetic honest. Sum of all pools across all PgBouncer instances must stay comfortably under PostgreSQL max_connections. Recheck this every time pool_size, instance count, or max_connections changes.
• Test the failover path. If your failover is DNS-based, verify that PgBouncer actually follows it: cache TTL, RELOAD behavior, and whether you need RECONNECT. Discovering stale DNS during a real failover is the worst time to learn this.
• Separate the timeout story. Know how your application’s statement/connection timeout interacts with query_wait_timeout. If the app times out at 10s and retries, a 30-second backend blip becomes a much longer retry-amplified event.

How Netdata helps

• Per-pool server connection states over time turn the draining signature into a visible trend: total server connections declining while sv_login stays elevated is the picture that separates backend failure from exhaustion at a glance.
• cl_waiting and maxwait per pool show queue depth and the oldest waiter’s age, so you can see how close clients are to the query_wait_timeout cliff.
• avg_wait_time versus avg_query_time correlation attributes latency correctly: wait time rising while query time stays flat means the backend path is broken, not the queries.
• Paused/disabled state collection lets alerts suppress themselves during maintenance instead of paging for an intentional PAUSE.
• Log-based alerting on connect failed and S: login failed closes the gap PgBouncer’s SHOW commands leave open, surfacing the root cause rather than only its queueing consequences.

Related guides

• PgBouncer how it works in production: a mental model for operators
• PgBouncer pool exhaustion: clients queue, wait times climb, and the retry cascade
• PgBouncer maxwait high: the oldest client waiter and how close it is to timing out
• PgBouncer avg_wait_time high: the latency the pool itself is injecting
• PgBouncer pool utilization high: sv_active approaching pool_size before clients queue
• PgBouncer capacity planning: runway for pools, clients, and PostgreSQL slots
• PgBouncer monitoring checklist: the signals every connection pooler needs
• PgBouncer monitoring maturity model: from survival to expert
• PgBouncer no more connections allowed (max_client_conn): the front door is full
• PgBouncer max_client_conn tuning: setting the client limit against real FD headroom
• PgBouncer advisory locks in transaction mode: orphaned locks and mysterious contention
• PgBouncer LISTEN/NOTIFY not working: why pub/sub needs session pooling