<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>PgBouncer Operations Guides on Netdata</title><link>https://www.netdata.cloud/guides/pgbouncer/</link><description>Recent content in PgBouncer Operations Guides on Netdata</description><generator>Hugo</generator><language>en-us</language><atom:link href="https://www.netdata.cloud/guides/pgbouncer/index.xml" rel="self" type="application/rss+xml"/><item><title>How PgBouncer actually works in production: a mental model for operators</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-how-it-works-in-production/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-how-it-works-in-production/</guid><description>&lt;p>Most PgBouncer incidents are the predictable consequence of a handful of internal mechanisms interacting under load: a single-threaded event loop, per-pool FIFO wait queues, fixed-size socket buffers, a small set of server connection states, and a pool mode that decides when server connections change hands. Hold these in your head and the alert thresholds stop being arbitrary numbers.&lt;/p>
&lt;p>This article is the model, not the triage. It explains what PgBouncer is doing internally so that when &lt;code>cl_waiting&lt;/code> spikes or &lt;code>sv_login&lt;/code> won&amp;rsquo;t drain, you already know which part of the machine is hurting and why.&lt;/p></description></item><item><title>PgBouncer advisory locks in transaction mode: orphaned locks and mysterious contention</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-advisory-locks-transaction-mode/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-advisory-locks-transaction-mode/</guid><description>&lt;p>Your application takes an advisory lock with &lt;code>pg_advisory_lock()&lt;/code>, does its work, releases it, and moves on. Except under load, other parts of the application start blocking on that same lock, or timing out waiting for it. The application is sure it released the lock. PostgreSQL&amp;rsquo;s lock views, looked at from the wrong place, show nothing. PgBouncer&amp;rsquo;s metrics are completely healthy: no waiting clients, no queue, normal wait times.&lt;/p>
&lt;p>This is not a saturation problem and not a bug in PgBouncer. It is a semantic mismatch between how advisory locks work in PostgreSQL and how transaction pooling reassigns server connections.&lt;/p></description></item><item><title>PgBouncer and PostgreSQL max_connections: when pool_size outruns the backend limit</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-postgresql-max-connections/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-postgresql-max-connections/</guid><description>&lt;p>PgBouncer exists to reduce the number of connections PostgreSQL has to hold, so it is easy to assume that adding PgBouncer makes the connection limit problem go away. It does not. PgBouncer is itself a consumer of PostgreSQL connection slots, and its worst-case demand is arithmetic you control in config files: one pool per (database, user) pair, each pool allowed to open pool_size server connections, plus reserve_pool_size overflow, multiplied by every PgBouncer instance pointing at the same backend.&lt;/p></description></item><item><title>PgBouncer auth failed / password authentication failed: client login rejected</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-auth-failed/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-auth-failed/</guid><description>&lt;p>PgBouncer logs &lt;code>closing because: auth failed&lt;/code> when a client fails to authenticate against the pooler itself. This is client-side authentication: the connection between your application and PgBouncer, not between PgBouncer and PostgreSQL. The connection is rejected before it enters any pool.&lt;/p>
&lt;p>These events are log-only. PgBouncer exposes no SHOW command counter for authentication failures. There is no &lt;code>auth_failed_count&lt;/code> in SHOW STATS, no per-user rejection tally, no rate metric. If you are not parsing the log, you are blind to auth failures.&lt;/p></description></item><item><title>PgBouncer auth_query failures: the authentication dependency loop on PostgreSQL</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-auth-query-failures/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-auth-query-failures/</guid><description>&lt;p>Clients cannot connect. The PgBouncer log fills with &amp;ldquo;password authentication failed&amp;rdquo; and &amp;ldquo;S: login failed&amp;rdquo; messages. &lt;code>SHOW POOLS&lt;/code> shows &lt;code>sv_login&lt;/code> connections stuck in the login state, &lt;code>cl_waiting&lt;/code> climbing. If you use &lt;code>auth_query&lt;/code> to authenticate clients against PostgreSQL, the root cause may be neither a wrong password nor a down database: it may be the circular dependency that &lt;code>auth_query&lt;/code> creates between the pooler and the backend.&lt;/p>
&lt;p>&lt;code>auth_query&lt;/code> tells PgBouncer to authenticate each connecting client by running a SQL query against PostgreSQL to look up that user&amp;rsquo;s password hash. You still need &lt;code>auth_file&lt;/code> (userlist.txt) for &lt;code>auth_user&lt;/code> credentials, but you do not need to sync every application user&amp;rsquo;s password into it. The cost: every new client connection requires a working PostgreSQL connection before the client can be pooled. When PostgreSQL is healthy, this is invisible. When PostgreSQL is under load, or when PgBouncer restarts and hundreds of clients re-authenticate simultaneously, &lt;code>auth_query&lt;/code> becomes the bottleneck that amplifies the outage.&lt;/p></description></item><item><title>PgBouncer avg_query_time high: reading backend slowdown through the pooler</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-avg-query-time-high/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-avg-query-time-high/</guid><description>&lt;p>&lt;code>avg_query_time&lt;/code> in PgBouncer&amp;rsquo;s &lt;code>SHOW STATS&lt;/code> output just doubled, and now &lt;code>cl_waiting&lt;/code> is starting to flicker above zero. This is the classic early-warning sequence for a pool exhaustion cascade: queries take longer, server connections are held longer, pool utilization climbs, and clients begin to queue. Catching the rise at the &lt;code>avg_query_time&lt;/code> stage, before &lt;code>cl_waiting&lt;/code> climbs, is the difference between a quiet Tuesday fix and a paged incident.&lt;/p>
&lt;p>The trap is that &lt;code>avg_query_time&lt;/code> is an average, and PgBouncer exposes no percentiles. A workload of 95% 1ms queries and 5% 500ms queries averages to a harmless-looking 26ms while the slow tail is already wrecking pool utilization. This article covers what the metric actually measures, how to diagnose a real rise versus an artifact, and how to trace the slowdown through to PostgreSQL.&lt;/p></description></item><item><title>PgBouncer avg_wait_time high: the latency the pool itself is injecting</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-avg-wait-time-high/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-avg-wait-time-high/</guid><description>&lt;p>Your application latency is up, PostgreSQL looks fine, and then PgBouncer&amp;rsquo;s stats show &lt;code>avg_wait_time&lt;/code> at tens or hundreds of milliseconds. That number answers &amp;ldquo;where did the latency come from&amp;rdquo;: it is time clients spent queued inside PgBouncer waiting for a server connection, before their query even reached PostgreSQL.&lt;/p>
&lt;p>&lt;code>avg_wait_time&lt;/code> is purely PgBouncer-induced latency. A direct connection to PostgreSQL would not have it. When it is high, the pool is not keeping up with demand, and every queued client pays that delay on top of normal query execution time.&lt;/p></description></item><item><title>PgBouncer backend unreachable: PostgreSQL down and the pool draining</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-backend-unreachable/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-backend-unreachable/</guid><description>&lt;p>Clients are queueing in PgBouncer, wait times are climbing, and &lt;code>SHOW POOLS&lt;/code> shows fewer server connections than there were an hour ago. Nobody changed the config. The likely cause: PostgreSQL is down, network-partitioned, or rejecting connections, and PgBouncer cannot establish new server connections to replace the ones it is losing.&lt;/p>
&lt;p>This failure mode is deceptive because it degrades slowly. Existing server connections keep serving queries until they expire (&lt;code>server_lifetime&lt;/code>, default 3600s), go idle past &lt;code>server_idle_timeout&lt;/code> (default 600s), or error out. The pool drains gradually rather than failing all at once. Meanwhile clients pile into the wait queue and eventually get disconnected at &lt;code>query_wait_timeout&lt;/code> (default 120s).&lt;/p></description></item><item><title>PgBouncer capacity planning: runway for pools, clients, and PostgreSQL slots</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-capacity-planning/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-capacity-planning/</guid><description>&lt;p>PgBouncer capacity planning fails in a specific way: operators size one resource, usually &lt;code>default_pool_size&lt;/code>, and forget the other three. Then a deployment doubles the app fleet, or a config change raises &lt;code>max_client_conn&lt;/code> without touching the OS file descriptor limit, and the first sign of trouble is an incident rather than a dashboard trend.&lt;/p>
&lt;p>There are four independent ceilings in any PgBouncer deployment, each with its own runway. The server pool determines how many queries can execute concurrently. Client slots determine how many application connections PgBouncer will accept. File descriptors determine what the operating system will let PgBouncer open. PostgreSQL backend slots determine whether the database has room for every pool PgBouncer might fill. Exhausting any one of them takes traffic down, and they degrade differently: some are cliffs, some are walls.&lt;/p></description></item><item><title>PgBouncer client connection leak: idle clients that never disconnect</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-client-connection-leak/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-client-connection-leak/</guid><description>&lt;p>&lt;code>used_clients&lt;/code> keeps climbing. Traffic is flat, &lt;code>cl_waiting&lt;/code> is zero, queries are fast, and yet the client count creeps toward &lt;code>max_client_conn&lt;/code> day after day. When it gets there, PgBouncer starts rejecting new connections with &lt;code>no more connections allowed (max_client_conn)&lt;/code>, even though the database itself is completely healthy.&lt;/p>
&lt;p>This is a client-side connection leak: application instances open connections to PgBouncer and never close them. PgBouncer is doing exactly what it is told to do, holding those sockets open. The pooler just makes the leak visible earlier and more painfully, because it has a hard front-door limit.&lt;/p></description></item><item><title>PgBouncer database paused or disabled: maintenance state that looks like an outage</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-database-paused/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-database-paused/</guid><description>&lt;p>&lt;code>cl_waiting&lt;/code> is climbing, &lt;code>sv_active&lt;/code> just dropped to zero, &lt;code>maxwait&lt;/code> is ticking upward. The pattern looks identical to pool exhaustion or a backend failure. But if someone is running planned PostgreSQL maintenance with &lt;code>PAUSE&lt;/code>, the metrics are behaving as designed: &lt;code>PAUSE&lt;/code> stops new query routing, existing transactions finish, server connections close, and every pending client queues. From the client&amp;rsquo;s perspective, it looks like an outage. The difference is that it is planned and reversible with &lt;code>RESUME&lt;/code>.&lt;/p></description></item><item><title>PgBouncer event loop stall: the single thread that freezes every pool at once</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-event-loop-stall/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-event-loop-stall/</guid><description>&lt;p>Every client connection, server connection, DNS lookup, and admin command runs on a single libevent thread inside PgBouncer. When something blocks that thread, every pool on every database freezes at the same time. Clients queue everywhere. The admin console itself becomes slow or unresponsive. This is not pool exhaustion; it is the entire process stalled.&lt;/p>
&lt;p>The signature is simultaneity. In normal pool exhaustion, one (database, user) pool saturates while others stay healthy. In an event loop stall, all pools degrade together, and the &lt;code>SHOW LISTS&lt;/code> command you run to diagnose the problem takes seconds to return. That admin console latency is the tell.&lt;/p></description></item><item><title>PgBouncer high CPU: single-core saturation, TLS, and connection churn</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-high-cpu/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-high-cpu/</guid><description>&lt;p>PgBouncer is consuming a disproportionate share of one CPU core. The process sits at 70%, 90%, or 100% of a single core while system-wide CPU looks normal. Application queries slow down and the admin console feels sluggish.&lt;/p>
&lt;p>PgBouncer is a single-threaded, event-driven process built on libevent. It runs on exactly one CPU core regardless of how many cores the machine has. A busy PgBouncer looks nearly idle on a multi-core box because system-wide CPU averages dilute the one saturated core. Measure per-process or per-core CPU, not system-wide averages.&lt;/p></description></item><item><title>PgBouncer idle in transaction: the silent pool killer in transaction mode</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-idle-in-transaction/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-idle-in-transaction/</guid><description>&lt;p>Your PgBouncer pool looks busy: &lt;code>sv_active&lt;/code> is pinned at &lt;code>pool_size&lt;/code>, &lt;code>cl_waiting&lt;/code> is climbing, and applications are timing out. But PostgreSQL is barely doing anything. CPU is low, &lt;code>avg_query_time&lt;/code> is a few milliseconds, and there are no slow queries to kill. The pool is full of connections that are &amp;ldquo;active&amp;rdquo; yet running nothing at all.&lt;/p>
&lt;p>This is the idle-in-transaction pattern, and it is the most common silent killer in transaction-mode PgBouncer deployments. An application runs &lt;code>BEGIN&lt;/code>, gets assigned a server connection, runs a query, then goes off to do non-database work (HTTP calls, serialization, computation, waiting on another service) before coming back to &lt;code>COMMIT&lt;/code>. In transaction pooling mode, that server connection stays assigned to the client for the entire transaction, including every second the client spends not talking to the database. Enough clients doing this and every server connection is checked out but idle. Everyone else queues.&lt;/p></description></item><item><title>PgBouncer LISTEN/NOTIFY not working: why pub/sub needs session pooling</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-listen-notify-transaction-mode/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-listen-notify-transaction-mode/</guid><description>&lt;p>Your application issues &lt;code>LISTEN job_events&lt;/code>, the command succeeds, and PostgreSQL&amp;rsquo;s logs show &lt;code>NOTIFY&lt;/code> firing on schedule. But the listener never receives anything. No error in the application. No error in PgBouncer. No metric anywhere that moves. The feature worked in staging, worked before you put PgBouncer in front of the database, and now it silently does nothing.&lt;/p>
&lt;p>This is the pool mode mismatch failure pattern, and LISTEN/NOTIFY is its most confusing variant because the failure is completely silent. Unlike prepared statements (which at least produce &amp;ldquo;prepared statement does not exist&amp;rdquo; errors), a lost LISTEN registration produces no error at all. The notification is delivered to a backend connection your client no longer holds, or to whichever client happens to hold that connection next.&lt;/p></description></item><item><title>PgBouncer max_client_conn tuning: setting the client limit against real FD headroom</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-max-client-conn-tuning/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-max-client-conn-tuning/</guid><description>&lt;p>Applications fail with connection errors, but PgBouncer&amp;rsquo;s pools look healthy: &lt;code>sv_active&lt;/code> is well below &lt;code>pool_size&lt;/code>, &lt;code>cl_waiting&lt;/code> is zero, and PostgreSQL is idle. The log tells the real story: &lt;code>no more connections allowed (max_client_conn)&lt;/code> or &lt;code>accept failed: Too many open files&lt;/code>. Clients are being refused before they ever reach a pool.&lt;/p>
&lt;p>The usual root cause is a mismatch between two limits operators treat as one. &lt;code>max_client_conn&lt;/code> is a configuration value. The OS file-descriptor limit (&lt;code>ulimit -n&lt;/code>) is a hard kernel ceiling. PgBouncer cannot accept more client connections than it has file descriptors for, no matter what the config says. If you set &lt;code>max_client_conn = 10000&lt;/code> while the process runs with the default 1024 FD limit, PgBouncer lowers the value at startup or hits the FD ceiling under load and refuses connections long before you expect it to. &lt;!-- TODO: verify whether the startup lowering is logged as a warning or fully silent in current PgBouncer versions -->&lt;/p></description></item><item><title>PgBouncer maxwait high: the oldest client waiter and how close it is to timing out</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-maxwait-high/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-maxwait-high/</guid><description>&lt;p>You opened &lt;code>SHOW POOLS&lt;/code> because an application is slow, and one column stands out: &lt;code>maxwait&lt;/code> is 8, 20, maybe 90 seconds. That number is the age of the oldest client sitting in PgBouncer&amp;rsquo;s FIFO wait queue, computed from the &lt;code>query_start&lt;/code> of the first waiter. It is the worst-case queuing latency any client is experiencing right now, before its query even reaches PostgreSQL.&lt;/p>
&lt;p>&lt;code>maxwait&lt;/code> is more actionable than the &lt;code>cl_waiting&lt;/code> count. &lt;code>cl_waiting&lt;/code> tells you how many clients are blocked; &lt;code>maxwait&lt;/code> tells you how much it hurts. A queue of 50 clients that drains in 200ms is a burst. A queue of 3 clients where the oldest has waited 45 seconds is an incident.&lt;/p></description></item><item><title>PgBouncer memory growth: RSS, pkt_buf, and the slab allocator</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-memory-growth/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-memory-growth/</guid><description>&lt;p>PgBouncer&amp;rsquo;s RSS is bounded by configuration: &lt;code>max_client_conn&lt;/code> plus the total server connection budget. The process pre-allocates connection structures at startup, so RSS stabilizes after warmup and should not grow monotonically under stable load. When it does, the cause is usually one of: &lt;code>pkt_buf&lt;/code> set too high, TLS session overhead, or a version-specific leak.&lt;/p>
&lt;h2 id="why-rss-matters-for-pgbouncer">Why RSS matters for PgBouncer&lt;/h2>
&lt;p>PgBouncer allocates memory proportional to connection count. The base cost is roughly 2KB per idle connection for socket buffer bookkeeping and connection metadata. For 10,000 clients, that is approximately 30-50MB of RSS at default settings. Active connections cost more because packet buffers are allocated to handle I/O.&lt;/p></description></item><item><title>PgBouncer monitoring checklist: the signals every connection pooler needs</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-monitoring-checklist/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-monitoring-checklist/</guid><description>&lt;p>PgBouncer is not a database. It is a single-threaded, event-driven proxy that multiplexes many client connections onto a smaller set of PostgreSQL connections, and it should be monitored the way you monitor HAProxy or nginx: queueing, connection exhaustion, and process health. The teams that get burned monitor PgBouncer with their PostgreSQL playbook (replication lag, WAL, bloat) and never check whether clients are actually waiting for connections.&lt;/p>
&lt;p>This checklist is organized by maturity level. Start at Level 1, get it alerting correctly, then work down. Every signal below comes from the admin console, the process table, or the PgBouncer log. There is no other source.&lt;/p></description></item><item><title>PgBouncer monitoring maturity model: from survival to expert</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-monitoring-maturity-model/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-monitoring-maturity-model/</guid><description>&lt;p>PgBouncer usually fails as a proxy, not as a process: it is alive, the port is open, the dashboards are green, and clients are still waiting two minutes for a server connection because nobody watched the wait queue. Its important failure modes are queueing, connection exhaustion, event loop stalls, and stale DNS. Most default database checks do not see them.&lt;/p>
&lt;p>This model has four levels, from &amp;ldquo;is it alive&amp;rdquo; to &amp;ldquo;correlate pool behavior with PostgreSQL and the application.&amp;rdquo; Each level answers a specific operational question. The goal is not to reach Level 4. The goal is to know which level you are actually at, and what you are blind to because of it.&lt;/p></description></item><item><title>PgBouncer no more connections allowed (max_client_conn): the front door is full</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-no-more-connections-allowed/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-no-more-connections-allowed/</guid><description>&lt;p>Your application logs fill with connection errors, and every new connection attempt to PgBouncer fails immediately with:&lt;/p>
&lt;pre tabindex="0">&lt;code>ERROR: no more connections allowed (max_client_conn)
&lt;/code>&lt;/pre>&lt;p>This is not pool exhaustion. The client never gets in the door. There is no queue, no wait, no &lt;code>query_wait_timeout&lt;/code>. PgBouncer counts the client connection, sees it would exceed &lt;code>max_client_conn&lt;/code>, and refuses it on the spot. Existing clients keep working; only new ones are turned away.&lt;/p>
&lt;p>The confusing part is that the pool itself may be completely healthy. &lt;code>sv_active&lt;/code> can be well below &lt;code>pool_size&lt;/code>, &lt;code>cl_waiting&lt;/code> can be zero, and &lt;code>avg_wait_time&lt;/code> can be flat. PostgreSQL is fine, the pools are fine, and yet applications cannot connect. The bottleneck is the front door, not the backend.&lt;/p></description></item><item><title>PgBouncer pool exhaustion: clients queue, wait times climb, and the retry cascade</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-pool-exhaustion/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-pool-exhaustion/</guid><description>&lt;p>Every server connection in a pool is busy. &lt;code>sv_active&lt;/code> equals &lt;code>pool_size&lt;/code>, &lt;code>sv_idle&lt;/code> is zero, and &lt;code>cl_waiting&lt;/code> is climbing. Clients that were getting sub-millisecond connection assignment are now sitting in a FIFO queue, and the oldest waiter (&lt;code>maxwait&lt;/code>) is old enough that application timeouts are firing. This is PgBouncer pool exhaustion, the most common PgBouncer incident, and it has a nasty property: it feeds itself.&lt;/p>
&lt;p>The cascade works like this. Server connections are held too long, so new client requests queue. Application-side timeouts are almost always shorter than PgBouncer&amp;rsquo;s &lt;code>query_wait_timeout&lt;/code> (default 120 seconds), so the application gives up first, retries, and adds a fresh waiter to the queue behind the one that never left. Retries arrive faster than the queue drains. &lt;code>cl_waiting&lt;/code> snowballs until &lt;code>max_client_conn&lt;/code> (default 100) is reached, at which point new connections are refused outright with &lt;code>&amp;quot;no more connections allowed (max_client_conn)&amp;quot;&lt;/code> in the log. What started as one slow query becomes a full outage.&lt;/p></description></item><item><title>PgBouncer pool utilization high: sv_active approaching pool_size before clients queue</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-pool-utilization-high/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-pool-utilization-high/</guid><description>&lt;p>Your alert fired on &lt;code>sv_active / pool_size &amp;gt; 85%&lt;/code>, or you spotted the ratio creeping up on a dashboard. No clients are waiting yet. &lt;code>cl_waiting&lt;/code> is zero. Latency looks normal. This is exactly the moment this signal exists for: it is the last cheap warning you get before the pool goes over the cliff.&lt;/p>
&lt;p>PgBouncer pool saturation is not a gradual degradation. Below 100% utilization, client wait time is approximately zero because server connection assignment is instant. At 100%, the next client request has nowhere to go and enters a FIFO queue with unbounded wait. There is no &amp;ldquo;slow but working&amp;rdquo; middle state. The ratio of &lt;code>sv_active&lt;/code> to &lt;code>pool_size&lt;/code> tells you how close you are to that edge, and it moves before &lt;code>cl_waiting&lt;/code>, &lt;code>maxwait&lt;/code>, and &lt;code>avg_wait_time&lt;/code> show anything.&lt;/p></description></item><item><title>PgBouncer pool_size sizing: matching pool capacity to transaction time and throughput</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-sizing-pool-size/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-sizing-pool-size/</guid><description>&lt;p>&lt;code>pool_size&lt;/code> decides how many queries can execute at once through PgBouncer. It is also the setting most teams guess at: leave the default, raise it when something breaks, lower it when PostgreSQL complains about connections. The correct value falls out of two numbers you can measure in a minute: peak transactions per second, and how long the average transaction holds a server connection.&lt;/p>
&lt;p>This article gives you the sizing relationship, shows how to measure its inputs from PgBouncer&amp;rsquo;s own stats, works through concrete examples, and covers the constraints that bound the answer: per-database overrides, cold start behavior, and the hard ceiling imposed by PostgreSQL&amp;rsquo;s &lt;code>max_connections&lt;/code>.&lt;/p></description></item><item><title>PgBouncer prepared statement does not exist: transaction pooling and lost session state</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-prepared-statement-does-not-exist/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-prepared-statement-does-not-exist/</guid><description>&lt;p>Your application starts throwing &lt;code>prepared statement &amp;quot;...&amp;quot; does not exist&lt;/code> errors in production. PgBouncer is healthy: no queuing, no wait time, pools well under capacity, PostgreSQL is fast. Single-user testing never reproduces it. Restarting the app makes it go away for a while, then it comes back under load.&lt;/p>
&lt;p>This is the pool mode mismatch failure pattern, and it is one of the nastier PgBouncer failure modes because every infrastructure signal looks green. The error is a SQL-level error from PostgreSQL, not a connectivity error from PgBouncer. Nothing in &lt;code>SHOW POOLS&lt;/code> or &lt;code>SHOW STATS&lt;/code> points at it. The only place it shows up is application error logs.&lt;/p></description></item><item><title>PgBouncer process down: the single point of failure in front of PostgreSQL</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-process-down/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-process-down/</guid><description>&lt;p>PgBouncer is one process on one host. Its single-threaded libevent loop handles every client socket, server socket, DNS lookup, and admin command on one CPU core. This gives it low overhead (roughly 2KB per idle connection) but makes it a hard single point of failure. When the PID disappears, the event loop stalls, or the process crash-loops, all database traffic through that instance is severed at once.&lt;/p>
&lt;p>Failure modes range from clean process death (OOM kill, segfault) to states where the process is alive, the port is open, but the event loop is frozen. A &lt;code>pgrep&lt;/code> check or port probe misses the latter. The correct health check is functional: connect to the admin console and run &lt;code>SHOW VERSION&lt;/code>. If it hangs or fails, PgBouncer is operationally down regardless of what systemd reports.&lt;/p></description></item><item><title>PgBouncer query rate drop: throughput falling without a traffic change</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-query-rate-drop/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-query-rate-drop/</guid><description>&lt;p>Your PgBouncer dashboard shows queries per second falling off a cliff, but the application team insists nothing changed: no deploy, no traffic shift, no feature flag. The &lt;code>avg_query_count&lt;/code> from &lt;code>SHOW STATS&lt;/code> (or the delta of &lt;code>total_query_count&lt;/code>) is down 40, 60, maybe 90 percent from baseline, and it is not coming back.&lt;/p>
&lt;p>A query rate drop with stable inbound traffic is almost never a PgBouncer bug. It is a symptom of something downstream or upstream: queries are taking longer so fewer complete per second, clients are queued instead of executing, the backend is unreachable, or the application itself has stopped sending work. The query rate is the smoke; your job is to find which fire is producing it.&lt;/p></description></item><item><title>PgBouncer query_wait_timeout: clients disconnected after waiting too long for a connection</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-query-wait-timeout/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-query-wait-timeout/</guid><description>&lt;p>Your PgBouncer log is filling with &lt;code>pooler error: query_wait_timeout&lt;/code> lines and application teams are reporting intermittent database errors. The error string sounds like a query problem. It is not. &lt;code>query_wait_timeout&lt;/code> fires when a client has been sitting in PgBouncer&amp;rsquo;s wait queue, blocked on getting a server connection, for longer than the configured timeout. The client never reached PostgreSQL.&lt;/p>
&lt;p>The default is 120 seconds. When this error appears, the pool has been exhausted for at least that long. &lt;code>query_wait_timeout&lt;/code> is a lagging indicator: it tells you an incident happened, not that one is starting.&lt;/p></description></item><item><title>PgBouncer reserve pool activation: overflow capacity that hides an undersized pool</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-reserve-pool-activation/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-reserve-pool-activation/</guid><description>&lt;p>PgBouncer&amp;rsquo;s reserve pool is overflow capacity: extra server connections beyond &lt;code>pool_size&lt;/code> that PgBouncer may open when clients have waited too long. Used as designed, it absorbs a short traffic spike and goes quiet. Used as a crutch, it hides a chronically undersized pool for months, until the day both base pool and reserve are exhausted and the queuing cliff is steeper than it would have been otherwise.&lt;/p>
&lt;p>This article covers the activation mechanism, how to detect reserve usage, how to tell healthy burst absorption from chronic undersizing, and what to change when it is the latter.&lt;/p></description></item><item><title>PgBouncer SCRAM / auth_type mismatch: md5 vs scram-sha-256 login failures</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-scram-auth-mismatch/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-scram-auth-mismatch/</guid><description>&lt;p>Clients cannot log in through PgBouncer. Authentication fails with errors that look like wrong passwords, but the passwords are correct and PostgreSQL accepts them on direct connection.&lt;/p>
&lt;p>The root cause is a mismatch between PgBouncer&amp;rsquo;s &lt;code>auth_type&lt;/code> and PostgreSQL&amp;rsquo;s &lt;code>password_encryption&lt;/code> or &lt;code>pg_hba.conf&lt;/code> method. PostgreSQL 12+ defaults to &lt;code>scram-sha-256&lt;/code> for &lt;code>password_encryption&lt;/code>. Deployments that upgrade PostgreSQL without updating PgBouncer&amp;rsquo;s auth configuration break silently because the authentication mechanisms are incompatible, not the credentials.&lt;/p>
&lt;p>This article covers how to identify the specific mismatch, align &lt;code>auth_type&lt;/code> with PostgreSQL&amp;rsquo;s password method, store SCRAM verifiers correctly, and migrate from md5 to scram without breaking existing connections.&lt;/p></description></item><item><title>PgBouncer server DNS lookup failed: stale cache and failed failover</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-server-dns-lookup-failed/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-server-dns-lookup-failed/</guid><description>&lt;p>You see &lt;code>server DNS lookup failed&lt;/code> in the PgBouncer log, usually right after a PostgreSQL failover, a DNS change, or a network event. New server connections cannot be established. Depending on timing, you may instead see the nastier variant: no error at all, just a pool quietly draining because PgBouncer&amp;rsquo;s DNS cache still points at the old primary IP.&lt;/p>
&lt;p>PgBouncer maintains its own DNS cache, independent of the TTL your DNS records publish. The cache lifetime is controlled by &lt;code>dns_max_ttl&lt;/code> (default: 15 seconds). After a failover, up to &lt;code>dns_max_ttl&lt;/code> can pass before PgBouncer even becomes eligible to re-resolve the backend hostname. Cached results are only re-queried when a new server connection is needed, so existing connections to the old IP keep running (against a dead or read-only host) while nothing forces a fresh lookup.&lt;/p></description></item><item><title>PgBouncer server login failed: PgBouncer cannot authenticate to PostgreSQL</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-server-login-failed/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-server-login-failed/</guid><description>&lt;p>Your PgBouncer log is filling with &lt;code>closing because: server login failed&lt;/code> (or &lt;code>server login timed out&lt;/code>), and application latency is climbing. PgBouncer accepts client connections fine. It can even reach PostgreSQL over the network. But every attempt to complete the backend login handshake fails, so no new server connections enter the pool.&lt;/p>
&lt;p>The symptom pattern is distinctive: &lt;code>sv_login&lt;/code> stays elevated while &lt;code>sv_idle&lt;/code> drains toward zero and &lt;code>cl_waiting&lt;/code> grows. Existing server connections keep working until they expire or are recycled, but nothing replaces them. The pool shrinks from the inside while clients pile up in the wait queue. If nothing is fixed, the pool empties completely and every client waits until &lt;code>query_wait_timeout&lt;/code> (default 120s) fires.&lt;/p></description></item><item><title>PgBouncer server_lifetime recycling waves: synchronized reconnects and capacity dips</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-server-lifetime-recycling/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-server-lifetime-recycling/</guid><description>&lt;p>Your dashboards show a repeating pattern: every hour (or whatever &lt;code>server_lifetime&lt;/code> is set to), &lt;code>sv_login&lt;/code> spikes, &lt;code>sv_idle&lt;/code> drops, and there is a brief bump in &lt;code>cl_waiting&lt;/code> or &lt;code>avg_wait_time&lt;/code>. It lasts seconds to a couple of minutes, then everything is green again. Application latency ticks up at the same moment. It looks like a flaky backend, but PostgreSQL is fine and the timing is suspiciously regular.&lt;/p>
&lt;p>This is the &lt;code>server_lifetime&lt;/code> recycling wave. PgBouncer closes each server connection once it has been connected longer than &lt;code>server_lifetime&lt;/code> (default 3600 seconds) and opens a replacement. If a large share of your server connections were created at roughly the same moment, which is exactly what happens after a PgBouncer restart, a PostgreSQL failover, or a cold start under load, they all hit the lifetime limit together and reconnect in lockstep. The pool briefly runs short on backend connections while the replacements authenticate.&lt;/p></description></item><item><title>PgBouncer server_reset_query and DISCARD ALL: the hidden per-return overhead</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-server-reset-query-discard-all/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-server-reset-query-discard-all/</guid><description>&lt;p>Every time PgBouncer returns a server connection to the pool in session pooling mode, it runs a cleanup query on that connection before anyone else can use it. By default that query is &lt;code>DISCARD ALL&lt;/code>, which wipes every piece of session state PostgreSQL is holding: temp tables, prepared statements, &lt;code>SET&lt;/code> variables, advisory locks, cursors, &lt;code>LISTEN&lt;/code> subscriptions. The reset is what makes connection sharing safe. It is also a backend round trip you pay for on every connection return, and it is rarely accounted for.&lt;/p></description></item><item><title>PgBouncer SET search_path lost between queries: session variables in transaction mode</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-set-search-path-lost/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-set-search-path-lost/</guid><description>&lt;p>Your application sets &lt;code>search_path&lt;/code> (or &lt;code>timezone&lt;/code>, or &lt;code>role&lt;/code>, or &lt;code>statement_timeout&lt;/code>) after connecting, and everything works in staging. In production, under concurrent load, queries intermittently hit the wrong schema, run with the wrong role context, or fail with &amp;ldquo;relation does not exist&amp;rdquo; for tables that clearly exist. Restarting the app &amp;ldquo;fixes&amp;rdquo; it briefly. Nothing in PgBouncer&amp;rsquo;s metrics looks wrong.&lt;/p>
&lt;p>This is pool mode mismatch: the application depends on session-level state, but PgBouncer is running in transaction pooling mode, where a client is assigned a different server connection for every transaction. Session state set on one server connection is not present on the next one. The failure is silent, load-dependent, and looks exactly like an application logic bug. PgBouncer itself reports nothing.&lt;/p></description></item><item><title>PgBouncer sv_idle at zero: no headroom and one slow query from a cascade</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-sv-idle-zero/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-sv-idle-zero/</guid><description>&lt;p>Your PgBouncer dashboards look green. &lt;code>cl_waiting&lt;/code> is zero, &lt;code>maxwait&lt;/code> is zero, no clients are queuing, no errors in the log. But &lt;code>SHOW POOLS&lt;/code> tells a different story: &lt;code>sv_idle&lt;/code> is 0 and &lt;code>sv_active&lt;/code> equals &lt;code>pool_size&lt;/code>. Every server connection in the pool is checked out. Nothing is waiting yet, but nothing is available either.&lt;/p>
&lt;p>This is the &amp;ldquo;looks green, is actually yellow&amp;rdquo; state, and it is one of the most dangerous steady states a connection pooler can sit in. The next request that arrives while all connections are busy queues immediately. There is no buffer, no graceful degradation. PgBouncer&amp;rsquo;s saturation curve is cliff-edge: below 100% utilization, assignment latency is effectively zero; at 100%, latency jumps to unbounded FIFO queuing.&lt;/p></description></item><item><title>PgBouncer thundering herd after restart: a login storm against PostgreSQL</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-thundering-herd-restart/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-thundering-herd-restart/</guid><description>&lt;p>PgBouncer just restarted. Maybe you pushed a config change that required it, maybe the process crashed, maybe the OOM killer took it out and systemd&amp;rsquo;s &lt;code>Restart=always&lt;/code> brought it right back. Within seconds, your dashboards light up: clients are queueing, wait times spike, and PostgreSQL is suddenly absorbing hundreds of simultaneous connection attempts. Then, usually, it calms down on its own within 10 to 60 seconds.&lt;/p>
&lt;p>This is a structural property of how PgBouncer works, not a bug. PgBouncer holds all of its server connections in process memory. When the process dies, every cached backend connection dies with it. The moment the new process starts accepting clients, every incoming query needs a brand new server connection, and every pool tries to build them all at once.&lt;/p></description></item><item><title>PgBouncer TLS: encrypted client and server connections, and certificate expiry</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-tls-configuration/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-tls-configuration/</guid><description>&lt;p>PgBouncer terminates or initiates TLS on two independent legs: the client leg (application to PgBouncer) and the server leg (PgBouncer to PostgreSQL). Each leg has its own configuration, certificate chain, and failure modes. Enabling TLS on one tells you nothing about the other.&lt;/p>
&lt;p>This independence is the source of most TLS operational surprises. A deployment can have fully encrypted client connections while the backend leg runs in plaintext, and nothing in PgBouncer&amp;rsquo;s metrics flags the discrepancy. The only reliable audit is the &lt;code>tls&lt;/code> column in &lt;code>SHOW CLIENTS&lt;/code> and &lt;code>SHOW SERVERS&lt;/code>.&lt;/p></description></item><item><title>PgBouncer Too many open files: file descriptor exhaustion and refused connections</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-too-many-open-files/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-too-many-open-files/</guid><description>&lt;p>PgBouncer is logging &lt;code>Too many open files&lt;/code> and clients are being turned away, or connections to PostgreSQL are failing with the same OS error. The confusing part: &lt;code>used_clients&lt;/code> is nowhere near &lt;code>max_client_conn&lt;/code>, the pools look healthy, and yet new connections are refused. The front door is not full. The kernel is out of file descriptors.&lt;/p>
&lt;p>PgBouncer is FD-hungry by design. Every proxied connection consumes roughly two file descriptors, one for the client socket and one for the server socket, plus a baseline of listening sockets, admin console sockets, DNS resolver sockets, and the log file. When the process hits its &lt;code>Max open files&lt;/code> limit, three things break at once: &lt;code>accept()&lt;/code> on the listen socket fails so new clients cannot connect, new server connections to PostgreSQL cannot be opened so existing clients start queuing, and log writes can fail so diagnostics disappear exactly when you need them.&lt;/p></description></item><item><title>PgBouncer transaction vs session pooling: what each mode breaks and when to use it</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-transaction-vs-session-pooling/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-transaction-vs-session-pooling/</guid><description>&lt;p>&lt;code>pool_mode&lt;/code> is the most consequential setting in &lt;code>pgbouncer.ini&lt;/code>. It decides when a server connection is returned to the pool, which in turn decides how many PostgreSQL backends you need and which PostgreSQL features your application is no longer allowed to use.&lt;/p>
&lt;p>The failure pattern that brings people to this page is consistent: someone switches from &lt;code>session&lt;/code> to &lt;code>transaction&lt;/code> for efficiency, deploys, and hours or days later the application starts throwing intermittent &amp;ldquo;prepared statement does not exist&amp;rdquo; errors, losing temp tables mid-request, or leaking advisory locks. Every PgBouncer metric looks healthy. The breakage only shows up under concurrency, because single-user testing keeps landing on the same backend.&lt;/p></description></item><item><title>PgBouncer wait time vs query time: is it the pool or the database?</title><link>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-wait-time-vs-query-time/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/pgbouncer/pgbouncer-wait-time-vs-query-time/</guid><description>&lt;p>Your application is slow. PgBouncer sits between the application and PostgreSQL, so the first question is always the same: is the latency coming from the pool itself, or from the database behind it? Operators routinely get this split wrong. They see high end-to-end latency, blame PostgreSQL, spend an hour in pg_stat_statements, and then discover avg_query_time was 5ms the whole time while avg_wait_time was 2000ms. The database was fast. The pool was too small.&lt;/p></description></item></channel></rss>