PgBouncer event loop stall: the single thread that freezes every pool at once

Every client connection, server connection, DNS lookup, and admin command runs on a single libevent thread inside PgBouncer. When something blocks that thread, every pool on every database freezes at the same time. Clients queue everywhere. The admin console itself becomes slow or unresponsive. This is not pool exhaustion; it is the entire process stalled.

The signature is simultaneity. In normal pool exhaustion, one (database, user) pool saturates while others stay healthy. In an event loop stall, all pools degrade together, and the SHOW LISTS command you run to diagnose the problem takes seconds to return. That admin console latency is the tell.

CPU behavior gives the second signal. A stalled PgBouncer process sits at one of two extremes: pinned at roughly 100% of one core (busy with TLS handshakes, verbose logging, or very high QPS), or near 0% CPU (blocked on synchronous I/O). Both produce the same external symptom: everything is slow.

Pool exhaustion vs event loop stall

PgBouncer is a single-threaded, event-driven multiplexer. The libevent loop handles all I/O: client sockets, server sockets, DNS lookups, admin console commands, and logging. One thread, one CPU core. This keeps overhead extremely low but means any synchronous operation that blocks the thread stalls every connection through every pool simultaneously.

The failure differs from pool exhaustion in both mechanism and scope:

• Pool exhaustion hits one pool. sv_active reaches pool_size for that (database, user) pair, sv_idle drops to zero, and cl_waiting grows for that pool only. Other pools remain unaffected. The admin console responds normally.
• Event loop stall hits everything. cl_waiting rises across all pools simultaneously, avg_wait_time spikes globally, and the admin console itself is slow. There is no single saturated pool to point at. The entire process is frozen or degraded.

Because the admin console runs on the same event loop, a trivial SHOW LISTS taking more than 1-2 seconds to return means the event loop is impaired. Admin console latency is the single best meta-health signal for PgBouncer.

flowchart TD
    A[All pools slow simultaneously] --> B{Admin console slow?}
    B -- No --> C[Single-pool issue:
check sv_active vs pool_size]
    B -- Yes, >1-2s --> D{PgBouncer CPU?}
    D -- ~100% of one core --> E[Busy stall:
TLS, verbose logging,
high QPS, small pkt_buf]
    D -- Near 0% --> F[Blocked stall:
sync DNS, stalled disk,
PAM queue full]

Common causes

Quick checks

These are safe, read-only commands. If the admin console is unresponsive, start with OS-level checks.

# Measure admin console latency - the primary meta-health signal
time psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW LISTS;" > /dev/null

# Check PgBouncer CPU - single-threaded, so one core matters
top -bn1 -p $(pgrep -o pgbouncer) | tail -1

# Check disk space on the log filesystem (adjust path to your logfile location)
df -h /var/log/pgbouncer/

# Check file descriptor usage - FD exhaustion can mimic a stall
ls /proc/$(pgrep -o pgbouncer)/fd | wc -l
grep "Max open files" /proc/$(pgrep -o pgbouncer)/limits

# Check all pools simultaneously - if admin console still responds
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW POOLS;"

# Check DNS resolution state and in-flight queries
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW DNS_HOSTS;"
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW LISTS;" | grep dns_queries

# Check for PAM queue warnings in logs (PgBouncer 1.25.0+)
grep -i "PAM queue" /var/log/pgbouncer/pgbouncer.log | tail -20

# Check which DNS backend PgBouncer was compiled with
ldd $(which pgbouncer) | grep -i "cares\|evdns\|udns"

# Check pkt_buf and sbuf_loopcnt configuration
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW CONFIG;" | grep -E "pkt_buf|sbuf_loopcnt"

How to diagnose it

Work through these steps in order. The goal is to classify the stall as “busy” (CPU at 100%) or “blocked” (CPU near 0%), then narrow to the specific cause.

Step 1: Confirm admin console latency. Run the timed SHOW LISTS. Under normal conditions, the response arrives in under 50ms. Anything above 200ms warrants investigation. Above 1-2 seconds is effectively a stall. If the command hangs entirely, the event loop is fully blocked.

Step 2: Classify by CPU. Check top for the PgBouncer process. Because it is single-threaded, look at one core, not system-wide CPU. Roughly 100% of one core means a busy stall: TLS, verbose logging, or very high query throughput. Near 0% means a blocked stall: the thread is waiting on synchronous I/O.

Step 3: For blocked stalls, check the log filesystem. Run df -h on the directory where PgBouncer writes its log. A full disk or hung NFS mount causes synchronous write() calls to block the event loop. Also check iostat for disk saturation on that device.

Step 4: For blocked stalls, check DNS. If the admin console responds, run SHOW DNS_HOSTS and look for entries with expired TTL or empty address lists. Check SHOW LISTS for dns_queries (in-flight DNS lookups). A consistently non-zero dns_queries indicates the resolver is slow or unreachable. If using the evdns2 backend and a nameserver has failed, DNS lookups may repeatedly fail until a process restart (GitHub issue #1119). Switching to c-ares is the recommended fix.

Step 5: For blocked stalls, check PAM authentication. If using PAM, check logs for “PAM queue is full” warnings (visible since PgBouncer 1.25.0). When the PAM request queue fills, PgBouncer calls usleep() on the main thread, blocking all I/O for the sleep duration. PgBouncer 1.25.2 still contains this code path.

Step 6: For busy stalls, check TLS. Review whether PgBouncer handles TLS termination directly. OpenSSL operations on the main thread can saturate the core during handshake storms, especially with slow clients or high connection churn. Correlate CPU spikes with new connection rates.

Step 7: For busy stalls, check logging verbosity. Higher verbose levels under high QPS generate logging overhead on the event loop. Check SHOW CONFIG for verbose and reduce it if necessary.

Step 8: Check pkt_buf and sbuf_loopcnt. A pkt_buf too small for result set sizes forces the event loop to iterate more times per socket, increasing CPU and syscall count. A sbuf_loopcnt of 0 removes the per-socket iteration limit, allowing one connection with a large result set to monopolize the loop. Verify sbuf_loopcnt is at its default of 5.

Step 9: Rule out SHOW FDS. SHOW FDS blocks the internal event loop. If any monitoring tool, debug script, or admin process calls it while PgBouncer handles traffic, the entire process stalls for the duration. Check admin console access logs.

Step 10: Differentiate from pool exhaustion. If only one pool shows cl_waiting > 0 and the admin console responds normally, the problem is pool exhaustion. See the PgBouncer avg_wait_time high guide.

Fixes

DNS resolution blocking

When PgBouncer cannot resolve backend hostnames asynchronously, DNS lookups block the event loop. The libc-based DNS backend is a blocking fallback that should never run in production.

• Verify c-ares is linked: ldd $(which pgbouncer) | grep cares. If absent, recompile or reinstall with c-ares support.
• Use IP addresses instead of hostnames in pgbouncer.ini for critical backends to bypass DNS entirely.
• After a failover, changing the backend host in pgbouncer.ini and running RELOAD marks that database dirty and opens new server connections with a fresh lookup. A RELOAD that leaves the host unchanged does not clear the existing DNS cache; wait for its TTL (default 15 seconds) or restart if an immediate refresh is required.
• If using the evdns2 backend and nameserver failures cause unrecoverable blocking, switch to c-ares.

Logging to a stalled or full disk

PgBouncer writes log entries synchronously from the event loop. If the log filesystem is full, the disk is saturated, or an NFS mount is hung, the write call blocks the entire process.

• Move the log file to local storage, not NFS.
• Ensure the log directory has adequate free space and alerting.
• Consider syslog (syslog = 1 in config) to move delivery to a syslog daemon. PgBouncer still makes the syslog() call from its event loop, so this reduces file I/O but is not a guaranteed non-blocking log path.
• Reduce verbose if verbose logging is consuming event loop cycles.

TLS and CPU saturation

OpenSSL operations on the main thread can saturate the single core during handshake storms.

• Offload TLS termination to a load balancer or sidecar proxy (HAProxy, Envoy, nginx) in front of PgBouncer. This drops CPU usage dramatically.
• If TLS must stay on PgBouncer, monitor TLS handshake rate and correlate with CPU.
• Use so_reuseport to run multiple PgBouncer processes sharing the same port, distributing connections across cores by the kernel.

pkt_buf and sbuf_loopcnt tuning

• Verify pkt_buf (default 4096 bytes) is adequate for your workload. A small pkt_buf with large result sets forces excessive event loop iterations and syscalls.
• Verify sbuf_loopcnt is at its default of 5. Setting it to 0 removes the per-socket iteration limit, allowing one connection to monopolize the event loop.
• pkt_buf requires a restart to take effect; sbuf_loopcnt can be changed with a reload.

PAM authentication queue saturation

When the PAM request queue fills (default size 20), PgBouncer calls usleep() on the main thread, blocking all I/O for approximately 100ms per occurrence. A flood of authentication requests can cause repeated blocking. The queue size and sleep duration are unchanged in PgBouncer 1.25.2.

• Monitor for “PAM queue is full” warnings in the log (visible since PgBouncer 1.25.0).
• If auth storms recur, consider a non-PAM auth method (trust, md5, scram-sha-256) that does not use the PAM queue.
• Rate-limit new connection attempts at the application or load balancer to reduce auth spikes.

SHOW FDS usage

SHOW FDS blocks the event loop. It is intended for PgBouncer’s internal FD transfer mechanism and should never be called while PgBouncer serves traffic.

• Audit any monitoring scripts, debug tools, or admin procedures that might call SHOW FDS.
• Ensure monitoring uses stats_users accounts (read-only SHOW access) rather than admin_users accounts.

Prevention

• Compile with c-ares. This is the recommended DNS backend. Avoid libc-based DNS, which falls back to blocking resolution.
• Keep logs on local disk. Never point PgBouncer’s log file at an NFS mount or a filesystem that can stall.
• Offload TLS when possible. TLS termination at a proxy in front of PgBouncer eliminates the biggest CPU consumer on the event loop.
• Monitor admin console latency continuously. A timed SHOW LISTS check is the cheapest and most reliable meta-health signal. Alert when it exceeds 1-2 seconds.
• Verify pkt_buf and sbuf_loopcnt after config changes. Ensure sbuf_loopcnt is never 0. Ensure pkt_buf matches your workload’s result set profile.
• Scale horizontally with so_reuseport. When single-core CPU becomes a bottleneck, run multiple PgBouncer processes sharing the same port. Each process has its own event loop and independent pools, so a stall in one does not freeze the others.
• Avoid SHOW FDS during operation. Audit tooling and scripts to ensure this command is never called on a live instance.
• Set realistic connection rate limits. Authentication storms (especially with PAM or auth_query) can block the event loop. Rate-limit new connections at the application or load balancer.

How Netdata helps

The key diagnostic signals for event loop stalls are timing-based and cross-correlational. Netdata’s per-second collection makes these patterns visible.

• Admin console latency as a metric: Netdata times the admin console response on every collection cycle, giving a continuous per-second view of event loop health rather than point-in-time snapshots.
• Per-process CPU at per-second resolution: Netdata tracks per-process CPU, making the single-core pattern (100% busy vs 0% blocked) immediately visible.
• Multi-pool correlation: When cl_waiting spikes across all pools simultaneously, Netdata’s dashboards make the simultaneity pattern obvious in a way that individual SHOW POOLS snapshots cannot.
• DNS state tracking: PgBouncer exposes dns_queries in SHOW LISTS; check it alongside monitoring data during a suspected resolver stall.
• File descriptor monitoring: Netdata tracks process FD counts against OS limits, catching FD exhaustion before it mimics a stall.
• ML anomaly detection: Sudden deviations in admin console latency, per-process CPU, or cross-pool wait time trigger anomaly alerts even when absolute thresholds are not breached.

Related guides

• PgBouncer advisory locks in transaction mode: orphaned locks and mysterious contention
• PgBouncer avg_query_time high: reading backend slowdown through the pooler
• PgBouncer avg_wait_time high: the latency the pool itself is injecting
• PgBouncer backend unreachable: PostgreSQL down and the pool draining
• PgBouncer capacity planning: runway for pools, clients, and PostgreSQL slots
• PgBouncer client connection leak: idle clients that never disconnect
• How PgBouncer actually works in production: a mental model for operators
• PgBouncer idle in transaction: the silent pool killer in transaction mode
• PgBouncer LISTEN/NOTIFY not working: why pub/sub needs session pooling
• PgBouncer max_client_conn tuning: setting the client limit against real FD headroom
• PgBouncer maxwait high: the oldest client waiter and how close it is to timing out
• PgBouncer monitoring checklist: the signals every connection pooler needs