$ guides / pgbouncer / pgbouncer-process-down ▌

Operations Guides

PgBouncer process down: the single point of failure in front of PostgreSQL

PgBouncer is one process on one host. Its single-threaded libevent loop handles every client socket, server socket, DNS lookup, and admin command on one CPU core. This gives it low overhead (roughly 2KB per idle connection) but makes it a hard single point of failure. When the PID disappears, the event loop stalls, or the process crash-loops, all database traffic through that instance is severed at once.

Failure modes range from clean process death (OOM kill, segfault) to states where the process is alive, the port is open, but the event loop is frozen. A pgrep check or port probe misses the latter. The correct health check is functional: connect to the admin console and run SHOW VERSION. If it hangs or fails, PgBouncer is operationally down regardless of what systemd reports.

Two failure states, different responses

There is no internal redundancy, no hot standby, no failover within the process. Two states look similar from the outside but require different responses:

Process gone: The PID no longer exists. If a supervisor is configured, it will attempt restart. The task is finding why it died.
Process alive, event loop frozen: The PID exists, the port shows LISTEN in ss, but no connections are serviced. systemctl is-active may report active. Only a functional check reveals the stall.

The event loop can stall without the process dying. Known causes include synchronous DNS resolution blocking the loop, logging to a full or stalled disk, a TLS handshake with a slow client monopolizing the thread, or the PAM authentication queue filling up.

flowchart TD
    A["PgBouncer appears down"] --> B{"pgrep finds PID?"}
    B -- "No" --> C{"dmesg shows OOM kill?"}
    C -- "Yes" --> D["OOM kill: check memory, pkt_buf, max_client_conn"]
    C -- "No" --> E["Check journalctl for segfault or crash"]
    B -- "Yes" --> F{"SHOW VERSION responds?"}
    F -- "Hangs or fails" --> G["Event loop frozen: check CPU, FDs, log disk"]
    F -- "Responds" --> H["Process healthy: check pools with SHOW POOLS"]
    G --> I["May require restart to recover"]
    D --> J["Fix root cause, restart, monitor warmup"]
    E --> J

Common causes

Cause	What it looks like	First thing to check
OOM kill	Process gone without crash trace in PgBouncer log	`dmesg \| grep -i oom`
Segfault or crash	Process gone, crash trace or signal in journal	`journalctl -u pgbouncer --since "30 min ago"`
Event loop stall	PID alive, port LISTEN, no connections serviced, SHOW VERSION hangs	`top -p $(pgrep pgbouncer)` for CPU; check log disk space
File descriptor exhaustion	New connections fail intermittently, process may crash-loop	`ls /proc/$PID/fd \| wc -l` vs FD limit
Config error on reload	Process exits or fails to start after RELOAD	`journalctl -u pgbouncer` for parse errors
Stale pidfile	Process gone, restart fails claiming pidfile exists	Check pidfile path, remove if stale
Remote crash (CVE)	Process dies on specific client input, no local resource pressure	Check PgBouncer version against latest advisories

Diagnose

All commands below are read-only and safe to run on a production host.

Step 1: Run the functional check first.

time psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW VERSION;"

If it responds in under 50ms, the process is healthy and the problem is elsewhere (pool exhaustion, backend unreachable, etc.). If it hangs or fails, continue.

Step 2: Determine whether the PID exists.

pgrep -f pgbouncer

No PID found: the process crashed or was killed. Go to step 3. PID exists but SHOW VERSION hangs: the event loop is stalled. Go to step 5.

Step 3: If the PID is gone, check for OOM kill.

dmesg | grep -i oom

The kernel OOM killer logs which process it killed and why. If PgBouncer was OOM-killed, check whether max_client_conn or pkt_buf is set too high for available memory, and whether memory pressure came from another process on the host.

Step 4: If no OOM kill, check for crash signals.

journalctl -u pgbouncer --since "30 min ago" --no-pager | tail -50

Look for segfault messages, signal numbers, or config parse errors. A config error on RELOAD (invalid value for a parameter) can cause the process to exit.

Step 5: If the PID exists but the event loop is stalled, check CPU.

top -bn1 -p $(pgrep pgbouncer)

CPU at or near 100% of one core: the event loop is saturated (TLS handshakes, excessive logging, busy loop). CPU near 0%: the thread is blocked on synchronous I/O (DNS, disk write for logging).

Step 6: Check file descriptor exhaustion.

PGBPID=$(pgrep -f pgbouncer)
echo "Open FDs: $(ls /proc/$PGBPID/fd | wc -l)"
grep "Max open files" /proc/$PGBPID/limits

FD exhaustion prevents new connections and can cause crash loops if PgBouncer cannot open its log file.

Step 7: Check the log disk.

If PgBouncer logs to a file and that filesystem is full or the mount is stalled (NFS hang, etc.), blocking writes freeze the event loop.

df -h

Step 8: After any restart, expect a thundering herd.

All server connections are lost on restart. Every client that sends a query triggers a new connection to PostgreSQL simultaneously. This usually self-resolves in 30-60 seconds, but if PostgreSQL’s max_connections is tight, the login storm can cause secondary failures. Set min_pool_size to pre-warm connections if the warmup window matters.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Functional health (SHOW VERSION latency)	Tests the actual event loop, not just process existence	Latency above 200ms or timeout
Process RSS vs expected footprint	Memory growth can lead to OOM kill	RSS significantly exceeding `max_client_conn * 4KB + overhead`
Open FD count vs limit	FD exhaustion prevents connections and can crash the process	Above 80% of Max open files
Process CPU (single core)	Single-threaded design means one core is the ceiling	Sustained above 70% of one core
Connection refusal rate in log	Only source of refusal data; SHOW commands have no error counters	Any sustained rate of “no more connections allowed”
Process restart count	Frequent restarts indicate crash loop or recurring OOM	Multiple restarts in an hour

Fixes

OOM kill

PgBouncer’s footprint is roughly 2KB per idle connection plus overhead, but TLS connections consume more (20-50KB per connection due to OpenSSL session state). Check pkt_buf (default 4096 bytes): setting it too large multiplies per-connection memory. Reduce max_client_conn to fit within available memory, increase host RAM, or move other processes off the host. If using TLS, account for the higher per-connection cost. Restart after fixing the root cause.

Event loop stall

If the process is alive but the event loop is frozen, the immediate fix is identifying and removing the blocking operation:

Blocking DNS: Ensure PgBouncer uses asynchronous DNS resolution. If backends are configured by IP address rather than hostname, DNS is bypassed entirely.
Full log disk: Free space on the log filesystem or redirect logging to a non-blocking destination.
TLS handshake saturation: If many simultaneous TLS handshakes are consuming the single thread, consider terminating TLS at a load balancer in front of PgBouncer.
PAM auth queue full: When the PAM authentication queue fills, PgBouncer reportedly calls usleep() and stops servicing the event loop. If using PAM auth, investigate auth throughput.

A restart clears the stall but does not prevent recurrence. If the process is unresponsive to normal signals, SIGQUIT forces immediate shutdown (all connections are dropped).

File descriptor exhaustion

Check the current limit: grep "Max open files" /proc/$(pgrep pgbouncer)/limits
Set LimitNOFILE in the systemd unit or increase ulimit -n for the PgBouncer user.
Validate that max_client_conn plus expected server connections, listening sockets, log FDs, pipe FDs, and admin sockets fit within the new limit with at least 20% headroom.
Restart PgBouncer for the new FD limit to take effect.

PgBouncer pre-allocates client structures at startup based on max_client_conn. The effective connection ceiling is the lower of max_client_conn and what the FD limit supports. If max_client_conn is set to 10,000 but the FD limit is 1,024, PgBouncer will accept roughly 500 clients before running out of file descriptors.

Config error on reload

If PgBouncer exits after a RELOAD due to a config error, check journalctl for the specific parse error. Fix the invalid value in pgbouncer.ini and restart. Some parameters require a full restart, not just RELOAD: listen_addr, listen_port, unix_socket_dir, and auth_type are among them. Verify changes took effect with SHOW CONFIG after restart.

Stale pidfile

After a crash, a stale pidfile can prevent PgBouncer from restarting. The process is gone but the file still contains the old PID. Remove the stale pidfile manually and restart.

Prevention

Process supervision with automatic restart. Configure systemd with Restart=on-failure or Restart=always. The upstream PgBouncer service unit uses Type=notify and Restart=on-failure.
Functional health checks, not process checks. A load balancer or monitoring system should probe SHOW VERSION or SHOW LISTS, not just check whether the port is open. A stalled event loop passes port checks and may even pass systemctl is-active.
Consider WatchdogSec with Type=notify. systemd’s notify watchdog can detect a hung process that has stopped sending heartbeats.
Multi-process deployment with so_reuseport. Running multiple PgBouncer processes on the same port distributes connections across cores and provides process-level redundancy. If one process crashes, the others continue serving. Monitoring must aggregate across all processes since each has independent pools and stats.
FD limit headroom. Set the FD limit to at least max_client_conn * 2 + 500 to account for server connections, DNS, logging, and overhead. Validate after every max_client_conn increase.
Memory headroom. Account for TLS overhead when calculating expected RSS. Monitor process memory and alert before OOM kill territory.
Keep PgBouncer updated. Remote crash vulnerabilities have been fixed in recent releases. A process that dies on specific client input with no local resource pressure may indicate an exploitable bug.

Monitoring with Netdata

Netdata’s per-second process metrics (CPU, memory, FDs) let you see resource pressure building before a crash or OOM kill. Process restart detection with per-second granularity correlates restarts with upstream changes, deployments, or load spikes. ML anomaly detection on process CPU and connection counts can flag event loop stalls (sudden CPU spike or drop to zero) before users report errors.

Configure a functional health check that probes SHOW VERSION latency alongside process metrics. The gap between “process exists” and “process is working” is where event loop stalls live, and correlating the two signals is how you catch them.