PgBouncer auth failed / password authentication failed: client login rejected

PgBouncer logs closing because: auth failed when a client fails to authenticate against the pooler itself. This is client-side authentication: the connection between your application and PgBouncer, not between PgBouncer and PostgreSQL. The connection is rejected before it enters any pool.

These events are log-only. PgBouncer exposes no SHOW command counter for authentication failures. There is no auth_failed_count in SHOW STATS, no per-user rejection tally, no rate metric. If you are not parsing the log, you are blind to auth failures.

The diagnostic fork is immediate. Occasional failures from known application IPs after a deploy almost always mean credential rotation that was not applied to userlist.txt and then not followed by a RELOAD. A sustained flood from unknown source IPs suggests brute-force probing. The first signal that tells you which one you are dealing with is source IP distribution in the log entries.

PgBouncer does not rate-limit auth failure logging. A brute-force attack can generate thousands of log lines per second and fill the disk if log rotation or external rate-limiting is not in place.

What this means

PgBouncer authenticates clients against its own auth_file (typically userlist.txt) or via auth_query, a SQL query run against the PostgreSQL backend. When authentication fails, PgBouncer rejects the connection and logs a message. The exact string depends on the authentication mechanism:

• password authentication failed - password-based auth (md5 or plain) where the password did not match
• SASL authentication failed - SCRAM-SHA-256 auth where the client proof was invalid
• certificate authentication failed - TLS certificate auth failure
• no such user / no such database - login rejected because the database or user is not configured in PgBouncer

The log_pooler_errors setting (default: 1) controls whether these error messages are logged. If set to 0, auth failures are silently dropped. The log_connections setting (default: 1) logs successful logins, not failures. No setting exposes failure counts as a metric.

Server-side auth failures (PgBouncer failing to authenticate to PostgreSQL) produce different log strings and are a separate problem from the client-side failures covered here.

flowchart TD
    A["Client connects to PgBouncer"] --> B{"auth_type = trust?"}
    B -- yes --> C["Auth bypassed: connection accepted"]
    B -- no --> D{"User in auth_file?"}
    D -- no --> E{"auth_user + auth_query set?"}
    E -- no --> F["login failed: user not found"]
    E -- yes --> G["Run auth_query on backend"]
    G --> H{"Query returns credentials?"}
    H -- no --> F
    H -- yes --> I{"Password hash matches?"}
    D -- yes --> I
    I -- no --> J["password/SASL auth failed"]
    I -- yes --> K["Client enters pool"]
    F --> L["Logged: closing because: auth failed"]
    J --> L

Common causes

Quick checks

# Check current auth_type
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW CONFIG;" | grep auth_type

# Check auth_file path
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW CONFIG;" | grep auth_file

# Count recent auth failures in the log
grep -c "auth failed\|password authentication failed\|SASL authentication failed" /var/log/pgbouncer/pgbouncer.log

# Recent failures with timestamps (last 50)
grep "auth failed\|password authentication failed\|SASL authentication failed" /var/log/pgbouncer/pgbouncer.log | tail -50

# Extract source IPs from failure entries to identify brute force (IPv4 only)
grep "auth failed\|login failed" /var/log/pgbouncer/pgbouncer.log | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | sort | uniq -c | sort -rn | head -20

# Check PgBouncer version
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW VERSION;"

# Check if auth_query is configured
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW CONFIG;" | grep auth_query

# Check if auth_user is configured (enables auth_query lookup path)
psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "SHOW CONFIG;" | grep auth_user

# Verify userlist.txt modification time vs last deploy
stat /etc/pgbouncer/userlist.txt

# Check disk space on log volume (brute force can fill it)
df -h /var/log/pgbouncer/

Paths shown (/var/log/pgbouncer/, /etc/pgbouncer/userlist.txt) are common defaults. Adjust for your installation.

How to diagnose it

Step 1: Determine the failure pattern. Extract source IPs, usernames, and timestamps from the log. The pattern tells you the category:

• Few IPs, known application hosts, started after a deploy: credential rotation issue.
• Many IPs or unknown IPs, diverse usernames, high rate: brute force or scanning.
• All connections for a specific user fail consistently: password format mismatch or wrong password.
• Failures correlated with server_lifetime recycling on version 1.25.0: SCRAM caching bug.

Step 2: Verify the auth_file is current. If credential rotation is the suspect, compare the password hash in userlist.txt against what PostgreSQL has stored:

-- On PostgreSQL, check the stored password format for the affected user
SELECT rolname, substr(rolpassword, 1, 20) AS password_prefix
FROM pg_authid WHERE rolname = 'affected_user';

If PostgreSQL stores a SCRAM secret (SCRAM-SHA-256$...) but userlist.txt has an MD5 hash (md5...), or vice versa, PgBouncer will error with cannot do SCRAM authentication: wrong password type. Both sides must use the same format, or the auth_file must contain plain-text passwords (which work with any password-based auth type).

Step 3: Check auth_type compatibility. The auth_type setting determines what PgBouncer expects from the client and what it looks for in auth_file:

• scram-sha-256: expects SCRAM secrets in auth_file. Clients must support SCRAM.
• md5: if auth_file contains a SCRAM secret for a user, SCRAM authentication is used automatically. MD5 hashes and plain-text passwords also work.
• trust: no authentication. Any client can connect with any password, including none. Never use in production.
• hba: uses an HBA file similar to PostgreSQL’s pg_hba.conf.

A mismatch between auth_type and the password format stored in auth_file causes consistent failures for all affected users, not intermittent ones.

Step 4: If using auth_query, verify the query and auth_user. When auth_user is set, PgBouncer looks up users not present in auth_file by running auth_query against the PostgreSQL backend. A failure here means the query returned no row for that user, or the password hash did not match.

Check that the auth_user has permission to read pg_authid and that the auth_query SQL is valid. The default auth_query (since PgBouncer 1.24.1) includes a VALID UNTIL check so expired passwords are rejected. On versions before 1.24.1, the default auth_query does not consider password expiration, and expired credentials may still authenticate.

Step 5: Check for version-specific bugs. If you are running PgBouncer 1.25.0, a known bug in ad-hoc SCRAM authentication caching causes password authentication failed errors after server connections are recycled via server_lifetime. The symptom is that failures begin after the first server_lifetime expiry and affect subsequent connections. Upgrading to 1.25.1 or later resolves this.

Step 6: Check for disk-full risk. If the failure rate is high (brute-force scenario), verify the log volume has not filled and PgBouncer can still write:

df -h /var/log/pgbouncer/
tail -1 /var/log/pgbouncer/pgbouncer.log

Metrics and signals to monitor

Fixes

Credential rotation not applied to auth_file

If passwords changed on PostgreSQL but userlist.txt was not updated:

1. Export current password hashes from PostgreSQL:

   SELECT rolname, rolpassword FROM pg_authid WHERE rolcanlogin;
   

2. Rebuild userlist.txt in the correct format: "username" "password_or_hash"
3. Apply the update: psql -h 127.0.0.1 -p 6432 -U pgbouncer pgbouncer -c "RELOAD;"

The RELOAD is required. Since PgBouncer 1.17.0, automatic auth_file reloading on file modification was removed. The file is only re-read on explicit RELOAD or process restart.

RELOAD applies the new auth_file immediately. Existing authenticated clients are unaffected. New connections with updated credentials succeed on the next attempt.

Brute-force attack

PgBouncer has no built-in rate-limiting for authentication failures. External controls are required:

• fail2ban or equivalent: Parse the PgBouncer log for auth failure patterns and ban source IPs after a threshold. A regex matching closing because: auth failed or password authentication failed with the client IP is sufficient.
• iptables or nftables rate limiting: Limit new connection attempts per source IP at the network layer.
• Network ACLs: If PgBouncer should only accept connections from known application subnets, enforce this at the firewall level. PgBouncer itself does not restrict source IPs unless auth_hba_file is configured.

If the log volume itself is the problem, configure external log rotation with size-based triggers, not just time-based rotation.

auth_type or password format mismatch

If the error is cannot do SCRAM authentication: wrong password type:

1. Determine what format PostgreSQL stores: SELECT rolname, substr(rolpassword, 1, 20) FROM pg_authid WHERE rolname = 'X';
2. Ensure auth_file uses the same format:
   • Copy the SCRAM secret from PostgreSQL directly into auth_file.
   • Use plain-text passwords in auth_file (works with any password-based auth_type).
   • Set auth_type = md5 and include SCRAM secrets in auth_file (SCRAM is used automatically).
3. RELOAD after changes.

auth_query failures

If users not in auth_file fail authentication:

1. Verify auth_user exists in auth_file with valid credentials.
2. Verify auth_user can execute the auth_query on PostgreSQL:

   -- Test the auth_query path as auth_user
   SET ROLE auth_user;
   SELECT rolname, rolpassword FROM pg_authid WHERE rolname = 'test_user' AND rolcanlogin;
   RESET ROLE;
   

3. If the default auth_query was customized, verify the SQL is valid and returns the expected columns.

SCRAM caching bug on PgBouncer 1.25.0

If running PgBouncer 1.25.0 and failures correlate with server_lifetime expiry:

• Upgrade to 1.25.1 or later.
• Temporary workaround: increase server_lifetime well beyond the recycling interval and schedule PgBouncer restarts during low-traffic windows. This reduces how often the bug triggers but does not eliminate it.

Prevention

• Credential rotation runbook: Every password change on PostgreSQL must include updating userlist.txt and issuing a RELOAD. Automate this step. The most common auth failure incident is a deploy that rotates database passwords but forgets the pooler.
• External rate-limiting: Deploy fail2ban or network-layer rate limiting before you need it. Without it, a brute-force attack fills the log disk before anyone notices.
• Log monitoring with alerting: Since no SHOW counter exists, implement log parsing that counts auth failures per time window and alerts on sustained rates. Alert differently for known-IP failures (likely credential issue) versus unknown-IP floods (likely attack).
• Never use auth_type = trust in production. It disables all authentication.
• Track PgBouncer version against known auth bugs. The 1.25.0 SCRAM caching bug and the pre-1.24.1 auth_query VALID UNTIL bypass are examples where version determines vulnerability.
• Validate auth_file format after generation. If auth_file is generated by a script, validate that the output format matches what auth_type expects before deploying.

How Netdata helps

• Log-based auth failure detection: PgBouncer exposes no SHOW counter for auth failures. Netdata’s log parser can extract and count auth failed and password authentication failed entries, providing a time-series signal where none exists natively.
• Source IP correlation: Correlating auth failure log patterns with SHOW CLIENTS connection data distinguishes credential misconfiguration (failures from known application IPs) from brute-force attacks (failures from unknown IPs) without manual log grepping.
• Disk usage monitoring on the log volume: Netdata tracks disk space per mount point with per-second resolution. Disk usage alerts fire before PgBouncer loses the ability to write.
• PgBouncer connection state context: Per-second collection of cl_waiting, client connection counts, and pool state shows whether auth failures are progressing to pool starvation or are contained at the authentication layer.
• Config change correlation: Netdata can surface when PgBouncer was restarted or reloaded, helping correlate the onset of auth failures with deployment or configuration events.

Related guides

• PgBouncer advisory locks in transaction mode: orphaned locks and mysterious contention
• PgBouncer avg_query_time high: reading backend slowdown through the pooler
• PgBouncer avg_wait_time high: the latency the pool itself is injecting
• PgBouncer backend unreachable: PostgreSQL down and the pool draining
• PgBouncer capacity planning: runway for pools, clients, and PostgreSQL slots
• PgBouncer client connection leak: idle clients that never disconnect
• PgBouncer database paused or disabled: maintenance state that looks like an outage
• PgBouncer event loop stall: the single thread that freezes every pool at once
• PgBouncer high CPU: single-core saturation, TLS, and connection churn
• How PgBouncer actually works in production: a mental model for operators
• PgBouncer idle in transaction: the silent pool killer in transaction mode
• PgBouncer LISTEN/NOTIFY not working: why pub/sub needs session pooling