PgBouncer monitoring maturity model: from survival to expert

PgBouncer usually fails as a proxy, not as a process: it is alive, the port is open, the dashboards are green, and clients are still waiting two minutes for a server connection because nobody watched the wait queue. Its important failure modes are queueing, connection exhaustion, event loop stalls, and stale DNS. Most default database checks do not see them.

This model has four levels, from “is it alive” to “correlate pool behavior with PostgreSQL and the application.” Each level answers a specific operational question. The goal is not to reach Level 4. The goal is to know which level you are actually at, and what you are blind to because of it.

All signals below come from the PgBouncer admin console, the PgBouncer log, or OS-level process inspection. One structural fact shapes everything: PgBouncer exposes no error counters through SHOW commands. Authentication failures, connection refusals, and timeout events exist only in the log. Any maturity level that ignores the log is blind to error conditions.

flowchart TD
  L1["Level 1 - Survival
Is it alive, is anyone blocked?"]
  L2["Level 2 - Operational
How full is the pool, how slow is the backend?"]
  L3["Level 3 - Mature
Why is it degrading, which pool, which connection?"]
  L4["Level 4 - Expert
How does PgBouncer behavior interact with PostgreSQL and the app?"]
  L1 --> L2 --> L3 --> L4

Level 1 - survival

The question: is PgBouncer accepting connections, and is anyone blocked right now?

Four signals. If you monitor nothing else, monitor these.

Functional liveness. A port check is not enough. A process can be running with a stalled event loop: the socket shows LISTEN but commands hang. psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW VERSION;" proves the event loop is processing commands. Use an admin or stats user that exists in your auth setup.
Client wait queue depth (cl_waiting). From SHOW POOLS, per (database, user) pool. This is the most important PgBouncer metric. Any sustained non-zero value means clients are experiencing latency injected by the pooler itself, and no error is raised until query_wait_timeout (default 120s) disconnects them. Teams that skip this signal miss connection starvation entirely.
Free client slots (free_clients). From SHOW LISTS. When this reaches zero, new connections are refused immediately. There is no queue and no degradation curve, just refusal.
Log tail for errors. Watch for authentication failed, no more connections allowed (max_client_conn), and timeout keywords such as query_wait_timeout, client_idle_timeout, and server_login_timeout. This is your only error channel.

Two checks prevent false escalations. First, check paused and disabled in SHOW DATABASES before treating high cl_waiting as an incident. A PAUSE during maintenance produces the same signature as an outage. Second, alert on sustained maxwait from SHOW POOLS, not raw cl_waiting > 0. Brief queueing during bursts is normal in transaction pooling mode; maxwait separates “acceptable spike” from “clients are stuck.”

You are ready for Level 2 when these signals catch impact but cannot explain it: you know clients are waiting, but not whether the cause is pool size, backend latency, or PgBouncer itself.

Level 2 - operational

The question: how full is the pool, and is the bottleneck PgBouncer or PostgreSQL?

Level 1 tells you there is impact. Level 2 tells you where it is coming from. The discipline is simple: never look at wait time without looking at query time. If avg_query_time is 5ms and avg_wait_time is 2000ms, PostgreSQL is fine and the pool is too small. Blaming the database in that situation wastes the incident.

Add these signals:

Pool utilization ratio (sv_active / pool_size). Per pool, from SHOW POOLS and SHOW DATABASES. Above 85% sustained is a capacity warning; at 100% the next request queues. The degradation curve is a cliff, not a slope: latency goes from roughly zero to unbounded at saturation.
Idle server connections (sv_idle). Your headroom. sv_idle = 0 with cl_waiting = 0 is the “looks green, is actually yellow” state: nobody is waiting yet, but the next slow query starts a cascade. Do not treat idle connections as waste and shrink pool_size; in transaction mode, idle server connections are the ready reserve.
Average wait time (avg_wait_time). From SHOW STATS_AVERAGES, in microseconds. The friction PgBouncer itself adds. Baseline it per pool; sustained elevation means routine saturation.
Average query time (avg_query_time). Backend responsiveness as seen through the pooler. A sustained 2x deviation from baseline points at PostgreSQL: slow queries, lock contention, or I/O.
Average transaction time (avg_xact_time). How long a server connection is held per transaction. This directly determines pool capacity: a pool with pool_size = 20 and avg_xact_time = 100ms sustains roughly 200 TPS. Double the transaction time and you halve capacity.
Query and transaction rates. Use the per-second averages in SHOW STATS_AVERAGES (avg_query_count, avg_xact_count) for baseline and context. The total_* counters in SHOW STATS are cumulative since process start and reset on restart, so alert on rates or computed deltas, not raw totals.
Process CPU. PgBouncer is single-threaded and SHOW commands do not expose CPU. Check per-process CPU from the OS. Typical usage is a few percent of one core; sustained high CPU means the event loop itself is the bottleneck and pool metrics become misleading.
File descriptor ratio. For the exact PgBouncer PID, compare ls /proc/$PID/fd | wc -l with Max open files in /proc/$PID/limits. FD exhaustion is a hard wall and often arrives before max_client_conn, because max_client_conn may be set without accounting for server connections, listen sockets, and log FDs.
DNS resolution state. SHOW DNS_HOSTS shows cached addresses and TTLs per backend hostname. Stale DNS after a failover silently prevents pool replenishment: existing connections keep working while new ones fail.

You are ready for Level 3 when you can see degradation coming but still answer “which pool, which connection, which client” only by manual spelunking during the incident.

Level 3 - mature

The question: why is it degrading, and can you catch it before users do?

Level 3 adds leading indicators and per-object visibility. Keep the Level 2 aggregates, but stop trusting aggregates alone: one saturated pool can hide behind nine healthy ones.

Server login queue (sv_login). Per pool, from SHOW POOLS. It should be zero or transiently low. A sustained or rising sv_login with dropping sv_idle means backend connections are failing to establish: PostgreSQL at max_connections, auth failures, network problems, or DNS problems. This separates “pool exhausted but refilling” from “pool draining and cannot refill.”
sv_used accumulation. Connections in sv_used are idle but were used before; if they sit idle longer than server_check_delay (default 30s), they must pass server_check_query before reuse. Persistent accumulation means the health-check pipeline is slowing reuse.
Admin console latency. Time a trivial command: time psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW LISTS;" > /dev/null. The admin console runs on the same event loop as client traffic, so this is the best meta-health signal PgBouncer offers. Compare against baseline; a sustained rise means the single thread is impaired and everything is affected.
Transaction-to-query time ratio. When avg_xact_time is much larger than avg_query_time, the gap is idle-in-transaction time: clients hold a server connection while doing application work. This is a common silent cause of pool exhaustion in transaction mode, and no single metric names it. The ratio does.
Per-pool breakdown. Track sv_active, cl_waiting, maxwait, and wait times per (database, user) pool, not just globally. Saturation is a per-pool phenomenon.
Per-connection request_time aging. SHOW SERVERS shows each server connection’s state and the timestamp of its latest request. During an exhaustion event, the active connection with the oldest request_time is the primary suspect; the link column traces it back to the client in SHOW CLIENTS.
Log-derived refusal and timeout rates. Count no more connections allowed, query_wait_timeout, client_idle_timeout, and server_login_timeout events per interval from the log. A query_wait_timeout event confirms pool exhaustion lasted the full timeout; a rising client_idle_timeout rate points at an application connection leak.

You are ready for Level 4 when the per-incident questions shift from “what is PgBouncer doing” to “how is PgBouncer interacting with PostgreSQL and the application.”

Level 4 - expert

The question: how does pooler behavior interact with the backend and the application over time?

These signals are usually added after repeated incidents, when the team learns that healthy-looking PgBouncer metrics can coexist with real problems at the boundaries.

Server lifetime recycling waves. A server connection becomes eligible for recycling when it is unused and older than server_lifetime; current PgBouncer staggers those disconnects by roughly server_lifetime / pool_size, while an active backend can remain in service beyond the threshold. Connections created together after a restart can still create correlated turnover and login activity, so track server connection creation rate over time.
Memory allocator state (SHOW MEM). Shows PgBouncer’s internal slab allocators. Rarely useful day to day; valuable for spotting unbounded growth in long-running instances. The output format is documented as subject to change, so treat parsing as version-fragile.
Wait time versus application P99. PgBouncer exposes averages, and averages hide bimodal behavior: mostly instant assignments plus a few very long waits can average to something that looks fine. Correlate avg_wait_time with application-side tail latency. If app P99 is bad while avg_wait_time is flat, queueing is bursty and PgBouncer’s rolling average over stats_period (default 60s) is smoothing it away.
Cross-correlation with pg_stat_activity. Match PgBouncer’s sv_active against PostgreSQL’s active and idle in transaction sessions. Mismatches indicate state inconsistency: connections PgBouncer thinks are busy that PostgreSQL sees as idle in transaction, or vice versa. There is no direct session-to-backend-PID mapping; correlation typically works by matching client addresses or by tracing SHOW SERVERS link relationships.
auth_query latency. With auth_query, the first login for a user/database triggers a credential lookup against PostgreSQL; PgBouncer stores the returned dynamic credentials in memory per database, so repeat logins with that cached credential need not repeat the query. A new user or a restart does. A slow or unreachable auth backend can therefore block first logins and pool assignment; watch login_clients in SHOW LISTS, sv_login in SHOW POOLS, and new-connection rate.

One portability note applies across all levels: SHOW STATS columns change between releases. Reference columns by name in collection code, never by position.

How Netdata helps

Per-pool wait and utilization together. Netdata collects cl_waiting, maxwait, and server connection states per pool from the admin console, so the Level 1 saturation signal and the Level 2 utilization ratio share one timeline without manual SHOW POOLS polling.
Wait time next to query time. The critical Level 2 distinction, pool too small vs database slow, is a visual correlation: avg_wait_time beside avg_query_time makes misattribution obvious.
Process-level context. Per-process CPU and file descriptor usage from the host appear alongside pool metrics, which Level 2 needs because PgBouncer exposes neither through SHOW commands.
Counter-reset handling. Rate charts are computed from deltas, so SHOW STATS reset-on-restart does not produce phantom traffic drops.
Anomaly context at higher levels. Anomaly flags on wait time, login queue depth, and transaction times can surface Level 3 and 4 deviations such as bursty queueing, recycling waves, and churn spikes that fixed thresholds miss.