PgBouncer thundering herd after restart: a login storm against PostgreSQL

PgBouncer just restarted. Maybe you pushed a config change that required it, maybe the process crashed, maybe the OOM killer took it out and systemd’s Restart=always brought it right back. Within seconds, your dashboards light up: clients are queueing, wait times spike, and PostgreSQL is suddenly absorbing hundreds of simultaneous connection attempts. Then, usually, it calms down on its own within 10 to 60 seconds.

This is a structural property of how PgBouncer works, not a bug. PgBouncer holds all of its server connections in process memory. When the process dies, every cached backend connection dies with it. The moment the new process starts accepting clients, every incoming query needs a brand new server connection, and every pool tries to build them all at once.

The good news: the pattern is self-resolving if PostgreSQL has the connection headroom to absorb the storm. The bad news: if you use auth_query, or if PostgreSQL’s max_connections is tight, the storm can tip into a feedback loop that does not resolve on its own.

What this means

Under steady state, a PgBouncer pool keeps a set of warm server connections (sv_idle) ready for immediate reuse. A client query gets an existing connection in microseconds. After a restart, every pool starts at zero server connections. The first queries against each pool trigger connection creation: TCP handshake, optional TLS negotiation, and PostgreSQL authentication for each one.

In SHOW POOLS, connections being established show up as sv_login. During a restart storm, sv_login spikes toward pool_size across all pools simultaneously, and cl_waiting spikes because no connections exist yet to serve queries. On the PostgreSQL side, pg_stat_activity shows a surge of new backends being created.

The simultaneous, all-pools nature is the signature. A normal pool exhaustion event hits one busy pool. A restart storm hits everything at once, right after a process start event.

flowchart TD
  A[PgBouncer restarts] --> B[All server connections lost]
  B --> C[Pools start empty: sv_idle = 0]
  C --> D[Every client query needs a new backend connection]
  D --> E[sv_login spikes to pool_size on all pools]
  D --> F[cl_waiting spikes: queries queue for connections]
  E --> G{PostgreSQL absorbs the login storm?}
  G -->|yes| H[Connections establish, drains in 10-60s]
  G -->|no: max_connections tight or auth_query contention| I[Login failures, retry loop, prolonged outage]
  F --> G

The auth_query case deserves special attention. When PgBouncer authenticates clients via auth_query, each new client login requires a query against PostgreSQL executed as auth_user. That query itself needs a server connection. During the bootstrap window, server connections are the scarcest resource in the system, and the auth queries compete for the same scarce new connections that client queries are waiting on. If PostgreSQL is also refusing or slowing new connections, you get a bootstrap feedback loop: clients cannot log in because auth queries cannot run, and auth queries cannot run because the pool is clogged with login attempts.

Common causes

Quick checks

All of these are read-only. Run them against the PgBouncer admin console and PostgreSQL itself.

# 1. Confirm the restart: process uptime and start time
ps -o pid,lstart,etime -p $(pgrep -f pgbouncer)

# 2. Check for an OOM kill as the trigger
dmesg | grep -i oom | tail -5
journalctl -u pgbouncer --since "-10 min" | tail -30

# 3. Pool states: sv_login and cl_waiting per pool, plus maxwait (oldest waiter)
psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -Atc "SHOW POOLS;"

# 4. Global counters: login_clients stuck authenticating with PgBouncer itself
psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -Atc "SHOW LISTS;"

# 5. PgBouncer log: login failures, connect failures, timeouts
tail -200 /var/log/pgbouncer/pgbouncer.log | grep -iE "login failed|connect failed|timeout"

The admin console socket path depends on your unix_socket_directories setting; use a TCP connection to the admin port if PgBouncer does not listen on a socket.

On the PostgreSQL side:

# 6. Is PostgreSQL absorbing the storm, or saturated?
psql -h <postgres-host> -U <user> -d postgres -Atc \
  "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

# 7. Headroom against the hard limit
psql -h <postgres-host> -U <user> -d postgres -Atc \
  "SELECT setting::int FROM pg_settings WHERE name = 'max_connections';"

How to diagnose it

1. Confirm the trigger. Correlate the storm onset with process start time. If uptime is seconds to minutes old when sv_login and cl_waiting spiked across all pools, you are looking at a restart storm, not a traffic event. SHOW STATS counters reset on restart, so any continuity-based monitor will show a discontinuity at the same moment.

2. Verify the all-pools signature. In SHOW POOLS, check that sv_login is elevated across many pools at once. A single hot pool points to a workload problem instead.

3. Check whether it is draining. Take SHOW POOLS snapshots 10 to 15 seconds apart. In the healthy case, sv_login converts to sv_idle and sv_active, and cl_waiting and maxwait trend down. The typical self-resolution window is 10 to 60 seconds.

4. If it is not draining, find out why logins fail. PgBouncer has no error counters in any SHOW command; failures are log-only. Look for login failed, connect failed, and timeout messages. Each failed server login triggers a backoff controlled by server_login_retry (default 15 seconds) before PgBouncer retries, during which new clients for that pool get errors. A storm that keeps failing looks like a sawtooth: attempt, fail, 15-second backoff, retry.

5. Check PostgreSQL’s headroom. Compare the current backend count against max_connections. If the sum of all PgBouncer pools (plus any direct clients and replication connections) exceeds what PostgreSQL can accept, the storm cannot succeed no matter how long you wait. Logins get refused and the retry loop continues.

6. If you use auth_query, check the auth path. During the storm, auth queries for new client logins need server connections from the same pools that are being rebuilt. If auth_user has no guaranteed connection slot, authentication itself starves. Symptoms: client login failures in the log while sv_login stays high and login_clients in SHOW LISTS climbs.

7. Rule out the impostors. Check SHOW DATABASES for paused or disabled flags before treating queueing as an incident; a PAUSE during maintenance produces identical-looking cl_waiting spikes. If existing connections work but new ones fail and the resolved backend address is stale, you may be looking at a DNS problem after a failover rather than a restart storm; check SHOW DNS_HOSTS.

Metrics and signals to monitor

Fixes

If it is draining on its own

Wait. The correct first response to a self-resolving restart storm is to do nothing for 30 to 60 seconds and watch cl_waiting and maxwait trend down. Killing connections or restarting again during the bootstrap window restarts the storm and usually makes it worse. If the storm resolves but keeps happening, the fix is to stop restarting so abruptly; see Prevention.

If PostgreSQL cannot absorb the storm

The total possible server connections across all PgBouncer pools (sum of every pool’s pool_size, plus reserve_pool_size if configured) must fit inside PostgreSQL’s max_connections with room left for superuser access, replication, and direct clients. If they do not, either raise max_connections on PostgreSQL or shrink the pools. As a working guideline, keep aggregate PgBouncer pool capacity under about 80 percent of max_connections. Multiple PgBouncer instances pointed at the same PostgreSQL multiply the demand, and server_login_retry (default 15s) means each failed login wave is followed by a 15-second backoff, so recovery from overshoot is slow.

If auth_query is starving itself

Give auth_user a guaranteed path to PostgreSQL. The standard mitigation is to reserve a dedicated connection slot for the auth user so auth queries never have to compete with the client login storm for backend connections. Without that reservation, the bootstrap feedback loop can keep the pool from ever filling.

Also note that auth_query behavior has version-specific security fixes (notably the VALID UNTIL handling fixed in 1.24.1 and the default auth_query hardening in 1.25.1). If you run a custom auth_query, keep it aligned with the current default’s semantics when you upgrade.

If the storm is caused by your restart procedure

Change the procedure, not the limits. See Prevention.

Prevention

• Rolling restart with so_reuseport. Run multiple PgBouncer processes sharing one listen port and restart them one at a time so the surviving processes keep serving with warm pools. Since 1.23, SIGTERM performs a graceful shutdown that waits for clients to disconnect, which makes drain-then-restart practical. The older --reboot online restart option is deprecated since 1.20; the supported replacement is so_reuseport plus rolling restarts.

• Pre-warm pools with min_pool_size. Setting min_pool_size above zero keeps a floor of server connections in a pool, which reduces cold-start latency for that pool. One caveat: the setting is enforced for pools that have a forced user or at least one connected client, so a purely on-demand pool may not warm up until its first client arrives. Pools with a forced user get the most benefit.

• Size PostgreSQL for the storm, not the average. The worst case is every pool rebuilding to pool_size simultaneously. Validate that the sum of all pools across all PgBouncer instances fits inside max_connections with margin, before your next restart proves it does not.

• Reserve an auth path. If you use auth_query, reserve a dedicated PostgreSQL connection slot for auth_user so authentication cannot starve during the bootstrap window.

• Tune expectations around server_login_retry. With the 15-second default, a failed first login wave means clients see errors for at least that long before the next attempt. Know this number when judging whether a storm is “stuck” or merely backing off.

• Stagger lifecycle recycling. The built-in spread of server_lifetime reconnects protects steady-state rotation, but it does nothing for a full restart. Do not rely on it for restart safety.

• Alert with context. Restart storms produce exactly the signals that pool-exhaustion alerts fire on: cl_waiting up, maxwait up. Suppress or annotate these alerts with process start events, and check paused/disabled state before paging, so planned restarts do not wake anyone up.

How Netdata helps

• Netdata collects PgBouncer pool-level metrics continuously, so sv_login, cl_waiting, and maxwait are captured as time series. You can see the storm’s shape: the simultaneous all-pools spike, and whether it is draining or stuck.
• Correlating sv_login against cl_waiting and avg_wait_time on one dashboard separates “connections being rebuilt” (recovering) from “connections failing to build” (logins erroring out).
• Process-level context such as uptime and restarts sits next to the pool metrics, making it easy to confirm the trigger: storm onset aligned with a process start is the thundering herd signature.
• PostgreSQL-side metrics let you watch backend count approach max_connections during the storm, which is the deciding factor between self-resolving and stuck.
• Because PgBouncer exposes no error counters via SHOW commands, pairing metrics with log monitoring for login failed and connect failed closes the biggest blind spot in this failure mode.

Related guides

• PgBouncer advisory locks in transaction mode: orphaned locks and mysterious contention
• PgBouncer avg_wait_time high: the latency the pool itself is injecting
• PgBouncer backend unreachable: PostgreSQL down and the pool draining
• PgBouncer capacity planning: runway for pools, clients, and PostgreSQL slots
• PgBouncer client connection leak: idle clients that never disconnect
• How PgBouncer actually works in production: a mental model for operators
• PgBouncer LISTEN/NOTIFY not working: why pub/sub needs session pooling
• PgBouncer max_client_conn tuning: setting the client limit against real FD headroom
• PgBouncer maxwait high: the oldest client waiter and how close it is to timing out
• PgBouncer monitoring checklist: the signals every connection pooler needs
• PgBouncer monitoring maturity model: from survival to expert
• PgBouncer no more connections allowed (max_client_conn): the front door is full