PgBouncer server DNS lookup failed: stale cache and failed failover

You see server DNS lookup failed in the PgBouncer log, usually right after a PostgreSQL failover, a DNS change, or a network event. New server connections cannot be established. Depending on timing, you may instead see the nastier variant: no error at all, just a pool quietly draining because PgBouncer’s DNS cache still points at the old primary IP.

PgBouncer maintains its own DNS cache, independent of the TTL your DNS records publish. The cache lifetime is controlled by dns_max_ttl (default: 15 seconds). After a failover, up to dns_max_ttl can pass before PgBouncer even becomes eligible to re-resolve the backend hostname. Cached results are only re-queried when a new server connection is needed, so existing connections to the old IP keep running (against a dead or read-only host) while nothing forces a fresh lookup.

This article covers how to confirm the stale-cache condition, how to force recovery, and how to tune DNS behavior so the next failover does not page you.

What this means

PgBouncer resolves backend hostnames asynchronously and caches the result. Two failure modes follow:

1. Lookup fails outright. The resolver returns an error or NXDOMAIN. PgBouncer logs server DNS lookup failed and cannot create new server connections. DNS errors and NXDOMAIN results are themselves cached for dns_nxdomain_ttl (default: 15 seconds), so a transient resolver hiccup suppresses retries for that window.
2. Lookup succeeds but returns the stale IP. The cache still holds the pre-failover address. New connections go to the old primary, which is dead, refusing connections, or demoted to standby and read-only. Applications see connection failures or read-only transaction errors. Connections established before the failover may keep working briefly, which masks the problem while the pool drains.

The second mode is the classic “failed failover” pattern: the database layer failed over correctly, but the pooler did not follow. PgBouncer has no health-based backend routing. It connects wherever the cached DNS answer says.

When a hostname’s resolution does change, existing server connections using the old address are closed as they are released back to the pool (timing depends on pooling mode), and new connections use the new address. Nothing force-closes active connections mid-transaction. That is what RECONNECT and RELOAD are for.

flowchart TD
  A[PostgreSQL failover - IP changes] --> B[DNS record updated]
  B --> C{PgBouncer DNS cache}
  C -->|stale entry within dns_max_ttl| D[old IP returned]
  C -->|lookup fails| E[server DNS lookup failed in log]
  D --> F[new connections hit dead or read-only host]
  E --> G[no new server connections]
  F --> H[pool drains, sv_idle falls]
  G --> H
  H --> I[cl_waiting grows until query_wait_timeout]
  J[RELOAD or RECONNECT] --> K[cache flushed, connections re-established]
  K --> C

Common causes

Quick checks

All checks run against the PgBouncer admin console and are read-only unless noted.

# See what IP PgBouncer has cached for each backend hostname,
# and how many seconds until the entry is eligible for re-query
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW DNS_HOSTS;"

# Compare against the current DNS answer from the same host
getent hosts pg-primary.example.com
# or
dig +short pg-primary.example.com

# Check whether DNS queries are stuck in flight
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW LISTS;" | grep dns

# Check pool state: is the pool draining?
# Falling sv_idle + rising sv_login + growing cl_waiting = connections not being replaced
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW POOLS;"

# Confirm the TTL and resolver config PgBouncer is running with
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW CONFIG;" | grep -i dns

# Look for the error and related connection failures in the log
grep -E "server DNS lookup failed|connect failed|login failed" /var/log/pgbouncer/pgbouncer.log | tail -30

Key things in SHOW DNS_HOSTS output: the resolved addresses per hostname, and the TTL column (seconds until the entry is eligible for re-lookup). If the address shown there does not match what dig returns from the same host, you have confirmed the stale-cache condition.

How to diagnose it

1. Confirm the symptom class. Grep the log for server DNS lookup failed. If present, resolution itself is failing. If absent but applications report connection errors or read-only errors after a failover, suspect the stale-IP variant instead.
2. Read the cache. Run SHOW DNS_HOSTS. Write down the resolved address for each backend hostname.
3. Read reality. Resolve the same hostname from the PgBouncer host (dig +short, getent hosts). Use the resolver PgBouncer is configured to use, not just the system default, if resolv_conf points elsewhere.
4. Compare. If the cached IP differs from live DNS, the cache is stale. If live DNS itself still returns the old IP, the problem is upstream of PgBouncer: the failover tooling did not update DNS, or the record TTL at your DNS layer is the bottleneck.
5. Check pool impact. Run SHOW POOLS. The signature of a DNS-blocked pool is total server connections declining over time (sv_idle falling, sv_login stuck or failing) while cl_waiting and maxwait grow. Distinguish this from ordinary pool exhaustion, where total connections stay pinned at pool_size. See PgBouncer backend unreachable: PostgreSQL down and the pool draining for the broader pattern.
6. Check for in-flight queries. SHOW LISTS and look at dns_queries. Consistently non-zero means the resolver is slow or unreachable, not just that the cache is stale.
7. Check both address families. If your DNS backend errors on AAAA queries, test explicitly: dig AAAA <hostname> @<resolver>. An AAAA error can fail the entire lookup even when the A record is fine.

Metrics and signals to monitor

Fixes

Wait out the TTL

If dns_max_ttl is at its 15-second default and the failover just happened, the cache entry expires on its own and the next needed server connection triggers a fresh lookup. Acceptable only if your RTO tolerates the wait plus connection drain time. The TTL expiring only makes the entry eligible for re-query. The actual re-query happens when a new server connection is needed, and existing connections to the old IP are closed as they are released, not immediately.

RELOAD to flush the DNS cache

# Disruptive: re-reads config and re-resolves all hostnames
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "RELOAD;"

RELOAD re-reads the configuration and forces re-resolution of backend hostnames. This is the standard recovery action when the cache is stale. It is not free: it re-processes the full config, and on versions before 1.24.0 it also recycled TLS connections, causing a noticeable reconnect wave. On 1.24.0 and later, TLS connections are only recycled if TLS settings actually changed.

RECONNECT to force new server connections

# Disruptive: closes all server connections for the database
# after they finish their current transaction
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "RECONNECT mydb;"
# or all databases
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "RECONNECT;"

RECONNECT (available since PgBouncer 1.11) closes server connections as they finish their current transaction, forcing replacements that re-resolve DNS. Use this when you have flushed or expired the cache but long-lived connections are still pinned to the old IP. In session pooling mode, connections are held for the whole client session, so drain time depends on client behavior. Expect a brief latency bump as new connections authenticate.

Fix the resolver path

If the lookup itself is failing rather than returning stale data: verify the resolver PgBouncer uses is reachable, check resolv_conf if you point PgBouncer at a non-default resolver file, and test AAAA behavior if your DNS backend is picky. A cached NXDOMAIN clears after dns_nxdomain_ttl (default 15s), or immediately on RELOAD.

Prevention

• Set dns_max_ttl deliberately. The 15-second default is reasonable for many setups, but if your failover mechanism is DNS-based, align dns_max_ttl with your RTO. Lower values speed failover at the cost of more resolver load, which is usually trivial for a handful of backend hostnames.
• Consider dns_zone_check_period. When set (default: 0, disabled), PgBouncer polls the zone SOA serial and re-queries hostnames when it changes, giving faster pickup than TTL expiry alone. It requires the c-ares DNS backend, and it does not help if your hostname is a CNAME chain into another zone, since the authoritative zone cannot be determined.
• Runbook the recovery commands. Failover automation should include RELOAD or targeted RECONNECT against PgBouncer as a post-promotion step, so recovery does not depend on someone remembering the cache exists.
• Prefer IPs or local resolution where DNS adds no value. If a backend address is effectively static, configuring an IP (or a managed /etc/hosts entry) bypasses the cache entirely. You trade failover flexibility for one less moving part.
• Test the failover path end to end. The database failover working is not the same as the pooled path recovering. Drill it: promote a replica, watch SHOW DNS_HOSTS, measure time-to-recovery through PgBouncer.
• Watch the signals, not just the log. Alert on draining pool totals plus rising cl_waiting during known failover windows, and check paused/disabled state before escalating, since maintenance produces lookalike symptoms.

How Netdata helps

• Netdata collects PgBouncer pool state (cl_waiting, maxwait, sv_active, sv_idle, sv_login) at per-second granularity, so the drain pattern of a stale-DNS event is visible as it develops, not after query_wait_timeout starts firing.
• Per-pool breakdowns show whether the problem is isolated to one (database, user) pool or hitting every backend, which separates a single stale hostname from a resolver-wide failure.
• Correlating sv_login against total server connections distinguishes “connections failing to establish” (DNS or backend problem) from “pool saturated but healthy” (ordinary exhaustion), the two most commonly confused PgBouncer incidents.
• High-resolution maxwait history tells you how close waiters came to query_wait_timeout during the event, which informs whether your dns_max_ttl and failover runbook actually meet your RTO.
• Log-based signals like server DNS lookup failed have no SHOW-command counter, so pairing PgBouncer metrics with host-level log monitoring closes the error-visibility gap.

Related guides

• PgBouncer advisory locks in transaction mode: orphaned locks and mysterious contention
• PgBouncer avg_wait_time high: the latency the pool itself is injecting
• PgBouncer backend unreachable: PostgreSQL down and the pool draining
• PgBouncer capacity planning: runway for pools, clients, and PostgreSQL slots
• PgBouncer client connection leak: idle clients that never disconnect
• How PgBouncer actually works in production: a mental model for operators
• PgBouncer LISTEN/NOTIFY not working: why pub/sub needs session pooling
• PgBouncer max_client_conn tuning: setting the client limit against real FD headroom
• PgBouncer maxwait high: the oldest client waiter and how close it is to timing out
• PgBouncer monitoring checklist: the signals every connection pooler needs
• PgBouncer monitoring maturity model: from survival to expert
• PgBouncer no more connections allowed (max_client_conn): the front door is full