PgBouncer query rate drop: throughput falling without a traffic change

Your PgBouncer dashboard shows queries per second falling off a cliff, but the application team insists nothing changed: no deploy, no traffic shift, no feature flag. The avg_query_count from SHOW STATS (or the delta of total_query_count) is down 40, 60, maybe 90 percent from baseline, and it is not coming back.

A query rate drop with stable inbound traffic is almost never a PgBouncer bug. It is a symptom of something downstream or upstream: queries are taking longer so fewer complete per second, clients are queued instead of executing, the backend is unreachable, or the application itself has stopped sending work. The query rate is the smoke; your job is to find which fire is producing it.

One trap before anything else: PgBouncer’s total_query_count and total_xact_count are cumulative counters that reset to zero on every restart. If your monitoring computes rates from raw totals without handling resets, a routine restart looks exactly like a traffic collapse. Rule that out first, because it is the cheapest check and the most embarrassing root cause.

What this means

PgBouncer counts every SQL statement it forwards in total_query_count (and completed transactions in total_xact_count), and exposes a rolling per-second average as avg_query_count over the stats period. These are throughput baselines, not error signals. PgBouncer has zero error counters in any SHOW command, so a rate drop is one of the few ways the pooler can tell you something is wrong.

The mechanics of a drop reduce to simple arithmetic. Throughput equals concurrent server connections divided by how long each is held. If avg_query_time or avg_xact_time doubles, the same pool completes half as many queries per second. If all server connections are busy and clients are stuck in the wait queue, executing throughput stays flat while demand grows, and per-client throughput collapses. If the application deadlocked or hung, nothing is being sent at all.

There is also the inverse case worth a mention: a sudden query rate spike with no traffic change usually means a retry storm (application timing out and resubmitting) or a cache-invalidation stampede. The diagnostic path below assumes a drop, but several of the same checks apply.

flowchart TD
  A[Query rate dropped] --> B{Counter reset?
uptime / process restart}
  B -->|yes| C[Restart artifact, not an incident]
  B -->|no| D{cl_waiting > 0?}
  D -->|yes| E{avg_query_time elevated?}
  E -->|yes| F[Backend slowdown
check PostgreSQL]
  E -->|no| G[Pool exhausted by long transactions
check idle-in-transaction]
  D -->|no| H{sv_login high or sv_idle draining?}
  H -->|yes| I[Backend connection failure
check PostgreSQL, DNS, network]
  H -->|no| J[Application stopped sending
check clients and app health]

Common causes

Quick checks

All commands are read-only. They run against the admin console; adjust host, port, and socket path to your deployment.

# 1. Current per-second rates and averages per database
psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -Atc "SHOW STATS_AVERAGES;"

# 2. Cumulative totals (watch for resets across samples)
psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -Atc "SHOW STATS;"

# 3. Pool state: cl_waiting, sv_active, sv_idle, sv_login, maxwait
psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -Atc "SHOW POOLS;"

# 4. Paused/disabled state and per-database connection limits
psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -Atc "SHOW DATABASES;"

# 5. Process uptime: was there a restart?
ps -o pid,etime,cmd -p $(pgrep -f pgbouncer)

# 6. Admin console responsiveness (event loop health)
time psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -c "SHOW LISTS;" > /dev/null

# 7. Timeout and connection errors (log-only; no SHOW counters exist)
grep -E "query_wait_timeout|connect failed|login failed|no more connections" /var/log/pgbouncer/pgbouncer.log | tail -30

# 8. DNS cache state for hostname-based backends
psql -h /var/run/postgresql -p 6432 -U pgbouncer pgbouncer -Atc "SHOW DNS_HOSTS;"

Notes on interpretation:

• SHOW STATS_AVERAGES smooths over the stats period, so a very recent drop may look muted. Take two SHOW STATS samples 30 seconds apart and compute the delta of total_query_count for a sharper reading.
• Column positions in SHOW STATS are version-dependent (1.23 added server_assignment_count). Reference columns by name or use SHOW STATS_AVERAGES.
• The server_check_query (default SELECT 1) is counted in the stats. With many pools this slightly inflates the floor of your query rate, so “zero” may read as a small nonzero number.

How to diagnose it

1. Rule out the counter reset. Check process uptime (check 5). If PgBouncer restarted at the moment the “drop” began, your monitoring misread a reset as a traffic loss. Fix the rate computation to handle resets and move on.

2. Rule out maintenance. Check SHOW DATABASES for paused = 1 or disabled = 1. During PAUSE, new queries queue and cl_waiting spikes while sv_active drains. This is expected behavior during operations like a PostgreSQL upgrade. Every other check below assumes the database is not paused.

3. Split wait time from query time. From SHOW STATS_AVERAGES, compare avg_wait_time against avg_query_time for the affected database. High wait time with normal query time means the pool itself is the bottleneck (go to step 4). High query time means PostgreSQL is slow and connections are held longer (go to step 5). Both near zero with a low query rate means nothing is arriving (go to step 6).

4. Confirm pool exhaustion. In SHOW POOLS, find the pool with cl_waiting > 0. Check whether sv_active equals pool_size and sv_idle is zero. Look at maxwait: if it is approaching query_wait_timeout (default 120s), clients are about to be disconnected and will likely retry, amplifying the problem. Then compare avg_xact_time to avg_query_time. A large gap means transactions are held open while the application does other work; confirm on PostgreSQL with pg_stat_activity filtered to state = 'idle in transaction'. Note that in transaction pooling mode, total_query_count counts individual statements, so a shift toward multi-statement transactions changes the ratio of queries to transactions without any real throughput change; check avg_xact_count alongside.

5. Chase the backend slowdown. If avg_query_time is elevated, the root cause is on PostgreSQL: lock contention, a missing index under a new query pattern, I/O saturation, vacuum pressure. PgBouncer cannot see per-query detail; use pg_stat_activity and pg_stat_statements on the backend. Also check sv_login in SHOW POOLS: if it is persistently high while total server connections decline, new backend connections are failing (PostgreSQL at max_connections, auth failure, network partition, stale DNS after failover). Correlate with SHOW DNS_HOSTS and the connect/login failure lines in the log.

6. Check the front door. If PgBouncer internals are all green, verify clients are actually connected and sending. SHOW CLIENTS shows per-client state; large numbers of idle clients with old connect_time and no recent request_time means the application is hung or deadlocked, not the pooler. Check application-side health, thread pools, and deploys. Also check used_clients from SHOW LISTS against max_client_conn: if new connections are being refused (log line “no more connections allowed”), some app instances may be down while others hold all the slots.

7. Check the pooler itself. If the admin console from check 6 in the quick checks was slow, the single-threaded event loop is saturated: everything degrades at once, across all pools. Look at the process CPU; one core pegged at 100% under high connection churn or TLS load is the classic shape. The fix is architectural (multiple processes with so_reuseport, or TLS offload), not a config tweak.

Metrics and signals to monitor

Fixes

Counter reset artifacts

Fix the monitoring, not PgBouncer. Use the avg_* columns or compute deltas of total_* between samples, and treat a decrease in a cumulative counter as a reset, not negative traffic. Tag the series with process start time so restarts are visible on dashboards.

Backend slowdown

This is a PostgreSQL incident that happens to be visible at the pooler. Identify the slow queries on the backend (pg_stat_activity, pg_stat_statements), cancel or terminate the worst offenders in coordination with the application team, and fix the underlying cause (index, lock, vacuum). Raising pool_size during a backend slowdown usually makes things worse: more concurrent queries on an already struggling database.

Pool exhaustion from long or idle transactions

Short term: identify the pool in SHOW POOLS, find the holding connections via SHOW SERVERS (old request_time in active state, link back to the client), and cancel the offenders on PostgreSQL. Medium term: fix the application pattern that holds transactions open across non-database work; on 1.25.0+ the transaction_timeout setting can bound this at the pooler. If the pool is genuinely undersized for legitimate load, increase pool_size (RELOAD applies it), but verify the total across all pools still fits within PostgreSQL max_connections with headroom.

Backend connection failure

Verify PostgreSQL is up and accepting connections directly from the PgBouncer host. Check whether PostgreSQL is at max_connections and rejecting PgBouncer’s logins. Check SHOW DNS_HOSTS for a stale address after failover; a RELOAD refreshes the DNS cache. Fix credential mismatches between PgBouncer’s auth config and PostgreSQL. Do not restart PgBouncer as a first move: a restart drops all server connections and triggers a login storm that can make recovery slower.

Application stall

If every PgBouncer signal is healthy and clients are connected but idle, escalate to the application side: hung workers, deadlocks, blocked event loops, or a dependency outage upstream of the app. From the pooler’s perspective there is nothing to fix; SHOW CLIENTS source addresses tell you which app instances stopped sending.

Event loop saturation

If the single core is the ceiling, the durable fixes are running multiple PgBouncer processes with so_reuseport, or moving TLS termination off the pooler. Reducing connection churn (saner application pool sizing, longer-lived connections) buys time.

Prevention

• Baseline the rate properly. Track avg_query_count and avg_xact_count per database as first-class metrics with rolling baselines, and alert on sustained deviation (for example >50% drop for several minutes), never on a single sample. Keep this as an investigation trigger, not a page by itself: throughput is context, and the playbook treats it as INFO-severity forensics.
• Always pair rate with attribution metrics. A rate-drop alert that does not also surface avg_wait_time, avg_query_time, and cl_waiting forces the on-call to rediscover the split every time. Put all four on one dashboard.
• Handle restarts in rate math. Counter resets are the number one false positive for this exact alert.
• Suppress during maintenance. Include paused/disabled state in every PgBouncer alert condition.
• Watch headroom, not just failure. sv_active / pool_size trending toward 85% and avg_wait_time creeping off zero are the early warnings that precede a throughput collapse. See PgBouncer capacity planning for runway estimation.
• Hunt idle-in-transaction continuously. Track the avg_xact_time to avg_query_time ratio; a widening gap is the most common silent precursor to pool exhaustion.

How Netdata helps

• Netdata collects PgBouncer admin-console stats continuously, so total_query_count deltas are computed with counter-reset handling instead of naive subtraction, eliminating the restart false positive.
• Query rate, transaction rate, wait time, and query time are charted together per database, so the attribution split (pool latency vs backend latency vs no traffic) is visible in one view instead of three ad-hoc psql runs.
• Per-pool cl_waiting, sv_active, sv_idle, and maxwait are tracked individually, which matters because one saturated pool hides behind healthy aggregates.
• Paused/disabled database state is collected alongside the traffic metrics, giving immediate context for maintenance-window drops.
• Process uptime and CPU per process are correlated on the same host view, so a restart or a single saturated core shows up next to the throughput change it caused.

Related guides

• PgBouncer avg_wait_time high: the latency the pool itself is injecting
• PgBouncer maxwait high: the oldest client waiter and how close it is to timing out
• PgBouncer backend unreachable: PostgreSQL down and the pool draining
• PgBouncer client connection leak: idle clients that never disconnect
• PgBouncer capacity planning: runway for pools, clients, and PostgreSQL slots
• PgBouncer monitoring checklist: the signals every connection pooler needs
• How PgBouncer actually works in production: a mental model for operators
• PgBouncer no more connections allowed (max_client_conn): the front door is full