Every client connection, server connection, DNS lookup, and admin command runs on a single libevent thread inside PgBouncer. When something blocks that thread, every pool on every database freezes at the same time. Clients queue everywhere. The admin console itself becomes slow or unresponsive. This is not pool exhaustion; it is the entire process stalled.

The signature is simultaneity. In normal pool exhaustion, one (database, user) pool saturates while others stay healthy. In an event loop stall, all pools degrade together, and the SHOW LISTS command you run to diagnose the problem takes seconds to return. That admin console latency is the tell.

CPU behavior gives the second signal. A stalled PgBouncer process sits at one of two extremes: pinned at roughly 100% of one core (busy with TLS handshakes, verbose logging, or very high QPS), or near 0% CPU (blocked on synchronous I/O). Both produce the same external symptom: everything is slow.

Pool exhaustion vs event loop stall

PgBouncer is a single-threaded, event-driven multiplexer. The libevent loop handles all I/O: client sockets, server sockets, DNS lookups, admin console commands, and logging. One thread, one CPU core. This keeps overhead extremely low but means any synchronous operation that blocks the thread stalls every connection through every pool simultaneously.

The failure differs from pool exhaustion in both mechanism and scope:

  • Pool exhaustion hits one pool. sv_active reaches pool_size for that (database, user) pair, sv_idle drops to zero, and cl_waiting grows for that pool only. Other pools remain unaffected. The admin console responds normally.
  • Event loop stall hits everything. cl_waiting rises across all pools simultaneously, avg_wait_time spikes globally, and the admin console itself is slow. There is no single saturated pool to point at. The entire process is frozen or degraded.

Because the admin console runs on the same event loop, a trivial SHOW LISTS taking more than 1-2 seconds to return means the event loop is impaired. Admin console latency is the single best meta-health signal for PgBouncer.

flowchart TD
    A[All pools slow simultaneously] --> B{Admin console slow?}
    B -- No --> C[Single-pool issue:
check sv_active vs pool_size]
    B -- Yes, >1-2s --> D{PgBouncer CPU?}
    D -- ~100% of one core --> E[Busy stall:
TLS, verbose logging,
high QPS, small pkt_buf]
    D -- Near 0% --> F[Blocked stall:
sync DNS, stalled disk,
PAM queue full]

Common causes

CauseWhat it looks likeFirst thing to check
Synchronous DNS resolutionCPU near 0%, all pools stall when new server connections are needed, sv_login stuckSHOW DNS_HOSTS for stale entries; verify DNS backend
Logging to stalled disk or NFSCPU near 0%, log writes block the loop, disk full or NFS mount hungdf -h on the log directory; check NFS mount health
TLS handshake with slow clientCPU at 100% or fluctuating, stalls correlate with connection spikesSHOW CLIENTS for connections stuck in login
pkt_buf too smallHigher CPU under large result sets, excessive syscalls streaming chunksSHOW CONFIG for pkt_buf value; correlate with bytes-sent spikes
sbuf_loopcnt set to 0One connection with a large result set monopolizes the loopSHOW CONFIG for sbuf_loopcnt; should default to 5
PAM auth queue saturationCPU near 0%, stalls during auth storms, “PAM queue is full” in logsCheck logs for PAM queue warnings (PgBouncer 1.25.0+)
SHOW FDS commandMomentary full stall when a tool or monitoring calls itCheck admin console access logs for SHOW FDS usage

Quick checks

These are safe, read-only commands. If the admin console is unresponsive, start with OS-level checks.

# Measure admin console latency - the primary meta-health signal
time psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW LISTS;" > /dev/null

# Check PgBouncer CPU - single-threaded, so one core matters
top -bn1 -p $(pgrep -o pgbouncer) | tail -1

# Check disk space on the log filesystem (adjust path to your logfile location)
df -h /var/log/pgbouncer/

# Check file descriptor usage - FD exhaustion can mimic a stall
ls /proc/$(pgrep -o pgbouncer)/fd | wc -l
grep "Max open files" /proc/$(pgrep -o pgbouncer)/limits

# Check all pools simultaneously - if admin console still responds
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW POOLS;"

# Check DNS resolution state and in-flight queries
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW DNS_HOSTS;"
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW LISTS;" | grep dns_queries

# Check for PAM queue warnings in logs (PgBouncer 1.25.0+)
grep -i "PAM queue" /var/log/pgbouncer/pgbouncer.log | tail -20

# Check which DNS backend PgBouncer was compiled with
ldd $(which pgbouncer) | grep -i "cares\|evdns\|udns"

# Check pkt_buf and sbuf_loopcnt configuration
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW CONFIG;" | grep -E "pkt_buf|sbuf_loopcnt"

How to diagnose it

Work through these steps in order. The goal is to classify the stall as “busy” (CPU at 100%) or “blocked” (CPU near 0%), then narrow to the specific cause.

Step 1: Confirm admin console latency. Run the timed SHOW LISTS. Under normal conditions, the response arrives in under 50ms. Anything above 200ms warrants investigation. Above 1-2 seconds is effectively a stall. If the command hangs entirely, the event loop is fully blocked.

Step 2: Classify by CPU. Check top for the PgBouncer process. Because it is single-threaded, look at one core, not system-wide CPU. Roughly 100% of one core means a busy stall: TLS, verbose logging, or very high query throughput. Near 0% means a blocked stall: the thread is waiting on synchronous I/O.

Step 3: For blocked stalls, check the log filesystem. Run df -h on the directory where PgBouncer writes its log. A full disk or hung NFS mount causes synchronous write() calls to block the event loop. Also check iostat for disk saturation on that device.

Step 4: For blocked stalls, check DNS. If the admin console responds, run SHOW DNS_HOSTS and look for entries with expired TTL or empty address lists. Check SHOW LISTS for dns_queries (in-flight DNS lookups). A consistently non-zero dns_queries indicates the resolver is slow or unreachable. If using the evdns2 backend and a nameserver has failed, PgBouncer may enter an unrecoverable blocked state (GitHub issue #1119).

Switching to c-ares is the recommended fix.

Step 5: For blocked stalls, check PAM authentication. If using PAM, check logs for “PAM queue is full” warnings (visible since PgBouncer 1.25.0). When the PAM request queue fills, PgBouncer calls usleep() on the main thread, blocking all I/O for the sleep duration.

Step 6: For busy stalls, check TLS. Review whether PgBouncer handles TLS termination directly. OpenSSL operations on the main thread can saturate the core during handshake storms, especially with slow clients or high connection churn. Correlate CPU spikes with new connection rates.

Step 7: For busy stalls, check logging verbosity. A log_level of debug under high QPS generates serialization overhead on the event loop. Check SHOW CONFIG for log_level and reduce if necessary.

Step 8: Check pkt_buf and sbuf_loopcnt. A pkt_buf too small for result set sizes forces the event loop to iterate more times per socket, increasing CPU and syscall count. A sbuf_loopcnt of 0 removes the per-socket iteration limit, allowing one connection with a large result set to monopolize the loop. Verify sbuf_loopcnt is at its default of 5.

Step 9: Rule out SHOW FDS. SHOW FDS blocks the internal event loop. If any monitoring tool, debug script, or admin process calls it while PgBouncer handles traffic, the entire process stalls for the duration. Check admin console access logs.

Step 10: Differentiate from pool exhaustion. If only one pool shows cl_waiting > 0 and the admin console responds normally, the problem is pool exhaustion. See the PgBouncer avg_wait_time high guide.

Fixes

DNS resolution blocking

When PgBouncer cannot resolve backend hostnames asynchronously, DNS lookups block the event loop. The libc-based DNS backend is a blocking fallback that should never run in production.

  • Verify c-ares is linked: ldd $(which pgbouncer) | grep cares. If absent, recompile or reinstall with c-ares support.
  • Use IP addresses instead of hostnames in pgbouncer.ini for critical backends to bypass DNS entirely.
  • If entries are stale after a failover, run RELOAD to force re-resolution.
  • If using the evdns2 backend and nameserver failures cause unrecoverable blocking, switch to c-ares.

Logging to a stalled or full disk

PgBouncer writes log entries synchronously from the event loop. If the log filesystem is full, the disk is saturated, or an NFS mount is hung, the write call blocks the entire process.

  • Move the log file to local storage, not NFS.
  • Ensure the log directory has adequate free space and alerting.
  • Consider switching to syslog (syslog = 1 in config) to delegate log I/O to the syslog daemon.
  • Reduce log_level from debug to info or warning if verbose logging is consuming event loop cycles.

TLS and CPU saturation

OpenSSL operations on the main thread can saturate the single core during handshake storms.

  • Offload TLS termination to a load balancer or sidecar proxy (HAProxy, Envoy, nginx) in front of PgBouncer. This drops CPU usage dramatically.
  • If TLS must stay on PgBouncer, monitor TLS handshake rate and correlate with CPU.
  • Use so_reuseport to run multiple PgBouncer processes sharing the same port, distributing connections across cores by the kernel.

pkt_buf and sbuf_loopcnt tuning

  • Verify pkt_buf (default 4096 bytes) is adequate for your workload. A small pkt_buf with large result sets forces excessive event loop iterations and syscalls.
  • Verify sbuf_loopcnt is at its default of 5. Setting it to 0 removes the per-socket iteration limit, allowing one connection to monopolize the event loop.
  • Both parameters require a restart to take effect.

PAM authentication queue saturation

When the PAM request queue fills (default size 20), PgBouncer calls usleep() on the main thread, blocking all I/O for approximately 100ms per occurrence. A flood of authentication requests can cause repeated blocking.

  • Monitor for “PAM queue is full” warnings in the log (visible since PgBouncer 1.25.0).
  • If auth storms recur, consider a non-PAM auth method (trust, md5, scram-sha-256) that does not use the PAM queue.
  • Rate-limit new connection attempts at the application or load balancer to reduce auth spikes.

SHOW FDS usage

SHOW FDS blocks the event loop. It is intended for PgBouncer’s internal FD transfer mechanism and should never be called while PgBouncer serves traffic.

  • Audit any monitoring scripts, debug tools, or admin procedures that might call SHOW FDS.
  • Ensure monitoring uses stats_users accounts (read-only SHOW access) rather than admin_users accounts.

Prevention

  • Compile with c-ares. This is the recommended DNS backend. Avoid libc-based DNS, which falls back to blocking resolution.
  • Keep logs on local disk. Never point PgBouncer’s log file at an NFS mount or a filesystem that can stall.
  • Offload TLS when possible. TLS termination at a proxy in front of PgBouncer eliminates the biggest CPU consumer on the event loop.
  • Monitor admin console latency continuously. A timed SHOW LISTS check is the cheapest and most reliable meta-health signal. Alert when it exceeds 1-2 seconds.
  • Verify pkt_buf and sbuf_loopcnt after config changes. Ensure sbuf_loopcnt is never 0. Ensure pkt_buf matches your workload’s result set profile.
  • Scale horizontally with so_reuseport. When single-core CPU becomes a bottleneck, run multiple PgBouncer processes sharing the same port. Each process has its own event loop and independent pools, so a stall in one does not freeze the others.
  • Avoid SHOW FDS during operation. Audit tooling and scripts to ensure this command is never called on a live instance.
  • Set realistic connection rate limits. Authentication storms (especially with PAM or auth_query) can block the event loop. Rate-limit new connections at the application or load balancer.

How Netdata helps

The key diagnostic signals for event loop stalls are timing-based and cross-correlational. Netdata’s per-second collection makes these patterns visible.

  • Admin console latency as a metric: Netdata times the admin console response on every collection cycle, giving a continuous per-second view of event loop health rather than point-in-time snapshots.
  • Per-process CPU at per-second resolution: Netdata tracks per-process CPU, making the single-core pattern (100% busy vs 0% blocked) immediately visible.
  • Multi-pool correlation: When cl_waiting spikes across all pools simultaneously, Netdata’s dashboards make the simultaneity pattern obvious in a way that individual SHOW POOLS snapshots cannot.
  • DNS state tracking: Netdata collects dns_queries from SHOW LISTS, surfacing resolver problems before they cascade.
  • File descriptor monitoring: Netdata tracks process FD counts against OS limits, catching FD exhaustion before it mimics a stall.
  • ML anomaly detection: Sudden deviations in admin console latency, per-process CPU, or cross-pool wait time trigger anomaly alerts even when absolute thresholds are not breached.