PgBouncer server DNS lookup failed: stale cache and failed failover

You see server DNS lookup failed in the PgBouncer log, usually right after a PostgreSQL failover, a DNS change, or a network event. New server connections cannot be established. Depending on timing, you may instead see the nastier variant: no error at all, just a pool quietly draining because PgBouncer’s DNS cache still points at the old primary IP.

PgBouncer maintains its own DNS cache, independent of the TTL your DNS records publish. The cache lifetime is controlled by dns_max_ttl (default: 15 seconds). After a failover, up to dns_max_ttl can pass before PgBouncer even becomes eligible to re-resolve the backend hostname. Cached results are only re-queried when a new server connection is needed, so existing connections to the old IP keep running (against a dead or read-only host) while nothing forces a fresh lookup.

This article covers how to confirm the stale-cache condition, how to force recovery, and how to tune DNS behavior so the next failover does not page you.

What this means

PgBouncer resolves backend hostnames asynchronously and caches the result. Two failure modes follow:

1. Lookup fails outright. The resolver returns an error or NXDOMAIN. PgBouncer logs server DNS lookup failed and cannot create new server connections. DNS errors and NXDOMAIN results are themselves cached for dns_nxdomain_ttl (default: 15 seconds), so a transient resolver hiccup suppresses retries for that window.
2. Lookup succeeds but returns the stale IP. The cache still holds the pre-failover address. New connections go to the old primary, which is dead, refusing connections, or demoted to standby and read-only. Applications see connection failures or read-only transaction errors. Connections established before the failover may keep working briefly, which masks the problem while the pool drains.

The second mode is the classic “failed failover” pattern: the database layer failed over correctly, but the pooler did not follow. PgBouncer has no health-based backend routing. It connects wherever the cached DNS answer says.

When a hostname’s resolution does change, existing server connections using the old address are closed as they are released back to the pool (timing depends on pooling mode), and new connections use the new address. Nothing force-closes active connections mid-transaction. RECONNECT (added in PgBouncer 1.9) and a connection-changing RELOAD mark the affected connections for replacement.

flowchart TD
  A[PostgreSQL failover - IP changes] --> B[DNS record updated]
  B --> C{PgBouncer DNS cache}
  C -->|stale entry within dns_max_ttl| D[old IP returned]
  C -->|lookup fails| E[server DNS lookup failed in log]
  D --> F[new connections hit dead or read-only host]
  E --> G[no new server connections]
  F --> H[pool drains, sv_idle falls]
  G --> H
  H --> I[cl_waiting grows until query_wait_timeout]
  J[connection-changing RELOAD or RECONNECT] --> K[connections marked for replacement, then re-established]
  K --> C

Common causes

Quick checks

All checks run against the PgBouncer admin console and are read-only unless noted.

# See what IP PgBouncer has cached for each backend hostname,
# and how many seconds until the entry is eligible for re-query
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW DNS_HOSTS;"

# Compare against the current DNS answer from the same host
getent hosts pg-primary.example.com
# or
dig +short pg-primary.example.com

# Check whether DNS queries are stuck in flight
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW LISTS;" | grep dns

# Check pool state: is the pool draining?
# Falling sv_idle + rising sv_login + growing cl_waiting = connections not being replaced
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW POOLS;"

# Confirm the TTL and resolver config PgBouncer is running with
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -Atc "SHOW CONFIG;" | grep -i dns

# Look for the error and related connection failures in the log
grep -E "server DNS lookup failed|connect failed|login failed" /var/log/pgbouncer/pgbouncer.log | tail -30

Key things in SHOW DNS_HOSTS output: the resolved addresses per hostname, and the TTL column (seconds until the entry is eligible for re-lookup). If the address shown there does not match what dig returns from the same host, you have confirmed the stale-cache condition.

How to diagnose it

1. Confirm the symptom class. Grep the log for server DNS lookup failed. If present, resolution itself is failing. If absent but applications report connection errors or read-only errors after a failover, suspect the stale-IP variant instead.
2. Read the cache. Run SHOW DNS_HOSTS. Write down the resolved address for each backend hostname.
3. Read reality. Resolve the same hostname from the PgBouncer host (dig +short, getent hosts). Use the resolver PgBouncer is configured to use, not just the system default, if resolv_conf points elsewhere.
4. Compare. If the cached IP differs from live DNS, the cache is stale. If live DNS itself still returns the old IP, the problem is upstream of PgBouncer: the failover tooling did not update DNS, or the record TTL at your DNS layer is the bottleneck.
5. Check pool impact. Run SHOW POOLS. The signature of a DNS-blocked pool is total server connections declining over time (sv_idle falling, sv_login stuck or failing) while cl_waiting and maxwait grow. Distinguish this from ordinary pool exhaustion, where total connections stay pinned at pool_size. See PgBouncer backend unreachable: PostgreSQL down and the pool draining for the broader pattern.
6. Check for in-flight queries. SHOW LISTS and look at dns_queries. Consistently non-zero means the resolver is slow or unreachable, not just that the cache is stale.
7. Check both address families. If your DNS backend errors on AAAA queries, test explicitly: dig AAAA <hostname> @<resolver>. An AAAA error can fail the entire lookup even when the A record is fine.

Metrics and signals to monitor

Fixes

Wait out the TTL

If dns_max_ttl is at its 15-second default and the failover just happened, the cache entry expires on its own and the next needed server connection triggers a fresh lookup. Acceptable only if your RTO tolerates the wait plus connection drain time. The TTL expiring only makes the entry eligible for re-query. The actual re-query happens when a new server connection is needed, and existing connections to the old IP are closed as they are released, not immediately.

RELOAD to flush the DNS cache

# Disruptive: re-reads config and re-resolves all hostnames
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "RELOAD;"

RELOAD re-reads the configuration and updates changeable settings. It does not provide a general DNS-cache flush. When a database definition’s connection parameters change, it marks that database’s existing server connections close_needed, and new connections use the changed settings; if the config is unchanged, the cache remains governed by its TTL. RELOAD also re-processes the full config. Before 1.24.0 it recycled TLS connections, causing a noticeable reconnect wave; 1.24.0 made that recycling conditional on a TLS-settings change.

RECONNECT to force new server connections

# Disruptive: closes all server connections for the database
# after they finish their current transaction
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "RECONNECT mydb;"
# or all databases
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "RECONNECT;"

RECONNECT (available since PgBouncer 1.9) closes server connections as they finish their current transaction and makes replacements. Use it after the cache has expired or after a connection-changing RELOAD, when long-lived connections are still pinned to the old IP. In session pooling mode, connections are held for the whole client session, so drain time depends on client behavior. Expect a brief latency bump as new connections authenticate.

Fix the resolver path

If the lookup itself is failing rather than returning stale data: verify the resolver PgBouncer uses is reachable, check resolv_conf if you point PgBouncer at a non-default resolver file, and test AAAA behavior if your DNS backend is picky. A cached NXDOMAIN clears after dns_nxdomain_ttl (default 15s), or after a reload that changes the affected database definition; RELOAD is not a general DNS-cache flush.

Prevention

• Set dns_max_ttl deliberately. The 15-second default is reasonable for many setups, but if your failover mechanism is DNS-based, align dns_max_ttl with your RTO. Lower values speed failover at the cost of more resolver load, which is usually trivial for a handful of backend hostnames.
• Consider dns_zone_check_period. When set (default: 0, disabled), PgBouncer polls the SOA serial of the zone it derives from the configured hostname and re-queries those hostnames when the serial changes, giving faster pickup than TTL expiry alone. It requires the c-ares DNS backend. Verify in your DNS service that its SOA serial changes on record updates and that SOA queries are permitted; do not assume operator-controlled serial behavior on managed DNS.
• Runbook the recovery commands. Failover automation should include a connection-changing RELOAD or targeted RECONNECT against PgBouncer as a post-promotion step, so recovery does not depend on someone remembering the cache exists.
• Prefer IPs or local resolution where DNS adds no value. If a backend address is effectively static, configuring an IP (or a managed /etc/hosts entry) bypasses the cache entirely. You trade failover flexibility for one less moving part.
• Test the failover path end to end. The database failover working is not the same as the pooled path recovering. Drill it: promote a replica, watch SHOW DNS_HOSTS, measure time-to-recovery through PgBouncer.
• Watch the signals, not just the log. Alert on draining pool totals plus rising cl_waiting during known failover windows, and check paused/disabled state before escalating, since maintenance produces lookalike symptoms.

How Netdata helps

• Netdata collects PgBouncer pool state (cl_waiting, maxwait, sv_active, sv_idle, sv_login) at per-second granularity, so the drain pattern of a stale-DNS event is visible as it develops, not after query_wait_timeout starts firing.
• Per-pool breakdowns show whether the problem is isolated to one (database, user) pool or hitting every backend, which separates a single stale hostname from a resolver-wide failure.
• Correlating sv_login against total server connections distinguishes “connections failing to establish” (DNS or backend problem) from “pool saturated but healthy” (ordinary exhaustion), the two most commonly confused PgBouncer incidents.
• High-resolution maxwait history tells you how close waiters came to query_wait_timeout during the event, which informs whether your dns_max_ttl and failover runbook actually meet your RTO.
• Log-based signals like server DNS lookup failed have no SHOW-command counter, so pairing PgBouncer metrics with host-level log monitoring closes the error-visibility gap.

Related guides

• PgBouncer advisory locks in transaction mode: orphaned locks and mysterious contention
• PgBouncer avg_wait_time high: the latency the pool itself is injecting
• PgBouncer backend unreachable: PostgreSQL down and the pool draining
• PgBouncer capacity planning: runway for pools, clients, and PostgreSQL slots
• PgBouncer client connection leak: idle clients that never disconnect
• How PgBouncer actually works in production: a mental model for operators
• PgBouncer LISTEN/NOTIFY not working: why pub/sub needs session pooling
• PgBouncer max_client_conn tuning: setting the client limit against real FD headroom
• PgBouncer maxwait high: the oldest client waiter and how close it is to timing out
• PgBouncer monitoring checklist: the signals every connection pooler needs
• PgBouncer monitoring maturity model: from survival to expert
• PgBouncer no more connections allowed (max_client_conn): the front door is full