PostgreSQL Operations Guides on Netdata

How PostgreSQL actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How PostgreSQL actually works in production: a mental model for operators

PostgreSQL is a reliable relational database, but that reliability is not magic. It is the emergent behavior of five concrete subsystems that interact in specific, observable ways: the Write-Ahead Log, Multi-Version Concurrency Control, checkpoints, autovacuum, and the process-per-connection model. If you do not understand how these interact, you will misdiagnose table bloat as a missing index, interpret checkpoint I/O spikes as disk failures, and respond to connection exhaustion by raising max_connections until the OOM killer arrives.

PgBouncer pool exhausted: how to diagnose and fix client waits

Mon, 01 Jan 0001 00:00:00 +0000

PgBouncer pool exhausted: how to diagnose and fix client waits

Connection timeouts appear in application logs, but pg_stat_activity shows plenty of idle PostgreSQL backends. The bottleneck is usually the connection pooler. When PgBouncer exhausts server connections, clients queue at the pooler instead of reaching PostgreSQL. Sustained queuing raises latency and can cause timeouts before the database sees the query.

Confirm pool exhaustion, distinguish it from PostgreSQL-side connection limits, and fix the root cause without restarting services.

PgBouncer vs Pgpool-II vs Odyssey: choosing a PostgreSQL connection pooler

Mon, 01 Jan 0001 00:00:00 +0000

PgBouncer vs Pgpool-II vs Odyssey: choosing a PostgreSQL connection pooler

PostgreSQL uses one backend process per connection. Past a few hundred concurrent connections, memory overhead and context switching dominate and throughput collapses. A connection pooler becomes architecturally mandatory at scale. PgBouncer, pgpool-II, and Odyssey make different trade-offs around session state, prepared statements, throughput, and operational complexity. The wrong choice leads to session-leak bugs, throughput ceilings, or split-brain failures unrelated to PostgreSQL itself.

PostgreSQL ALTER TABLE blocked: zero-downtime DDL patterns

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL ALTER TABLE blocked: zero-downtime DDL patterns

An ALTER TABLE to add a column or change a type hangs. Application queries time out. Connection pools saturate. What looked like a simple schema change becomes a production incident.

By default, ALTER TABLE acquires an ACCESS EXCLUSIVE lock. It conflicts with every other lock mode, including ACCESS SHARE held by a plain SELECT. Once the DDL statement queues behind a blocker, every subsequent read and write on that table queues behind the waiting DDL. The table goes offline before the ALTER TABLE executes any work.

PostgreSQL autovacuum blocked by long-running transaction: detection and fix

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL autovacuum blocked by long-running transaction: detection and fix

Autovacuum workers are active, but n_dead_tup climbs and queries slow down as they scan dead tuples. Table bloat grows because VACUUM cannot reclaim dead row versions: a long-running transaction, abandoned replication slot, or hot-standby feedback is pinning the xmin horizon cluster-wide. The worker is not broken; it is blocked by an older transaction ID that must remain visible.

What this means

PostgreSQL’s MVCC keeps old tuple versions in the table until VACUUM removes them. VACUUM cannot remove any tuple that might still be visible to an active transaction. The boundary is the xmin horizon: the oldest transaction ID still active anywhere in the cluster.

PostgreSQL autovacuum not running: detection, causes, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL autovacuum not running: detection, causes, and fixes

Table sizes grow faster than insert rates. Query latency creeps up on UPDATE-heavy workloads. pg_stat_user_tables.n_dead_tup climbs while last_autovacuum stays frozen. Autovacuum should clean this up, but it is not. Dead tuple accumulation degrades performance and can eventually trigger transaction-ID-wraparound shutdown. Detect why autovacuum is stalled, find the blocker, and fix it without making things worse.

What this means

Autovacuum is a background subsystem that spawns workers to run VACUUM and ANALYZE based on table-level thresholds. When it works, it reclaims dead tuple space, updates the free space map, maintains the visibility map, and freezes old transaction IDs to prevent wraparound. When it stops, effects cascade: table and index bloat grow, sequential scans read more dead pages, the planner chooses worse plans as statistics stale, and database age advances toward the 2-billion-transaction hard limit.

PostgreSQL autovacuum tuning: per-table thresholds for high-churn workloads

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL autovacuum tuning: per-table thresholds for high-churn workloads

Default autovacuum settings target modest OLTP workloads. On a 500-million-row table, the global autovacuum_vacuum_scale_factor of 0.2 means autovacuum ignores the table until dead tuples exceed twenty percent of the row count. For high-churn tables, that delay lets bloat accumulate, degrades indexes, slows scans, and pushes transaction ID age toward wraparound. PostgreSQL overrides these thresholds per table via storage parameters (reloptions), so a 50 GB hot table can run aggressive settings without saturating workers across a 500 MB reference table. This guide covers calculating overrides, sizing worker memory, and verifying vacuum keeps up without drowning the cluster in background I/O.

PostgreSQL backup strategy: pg_dump, pg_basebackup, and pgBackRest compared

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL backup strategy: pg_dump, pg_basebackup, and pgBackRest compared

Your backup tool determines your RPO, RTO, and whether you can restore to a point in time or only to the backup moment. This guide compares logical dumps via pg_dump, built-in physical copies via pg_basebackup, and the third-party tool pgBackRest. PostgreSQL 17 introduced native incremental physical backups. Operators must decide whether built-in capabilities are sufficient or whether to adopt alternatives such as Barman or WAL-G.

PostgreSQL blocking queries: finding the root blocker in a lock cascade

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL blocking queries: finding the root blocker in a lock cascade

A query that normally finishes in milliseconds is now running for minutes. pg_stat_activity shows a queue of sessions with wait_event_type = 'Lock'. You identify one session holding the contested lock, but terminating it does not clear the queue. That session was itself blocked by another, which was blocked by another. Until you find the session at the head of the chain, the cascade continues.

PostgreSQL checkpoint storms: detection, causes, and tuning

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL checkpoint storms: detection, causes, and tuning

If query latency spikes and TPS drops coincide with your checkpoint_timeout schedule or follow a large bulk load, you are likely hitting a checkpoint storm. PostgreSQL checkpoints guarantee that dirty buffers are on disk. In a healthy system, timed checkpoints occur at checkpoint_timeout intervals and the background writer spreads that I/O. A storm happens when WAL generation hits max_wal_size before the timeout, forcing a requested checkpoint that flushes most dirty buffers at once, saturating disk bandwidth and stalling queries.

PostgreSQL connection exhaustion: detection, diagnosis, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL connection exhaustion: detection, diagnosis, and prevention

Application logs show FATAL: sorry, too many clients already. Health checks are failing. A rolling deploy just finished, and now the database is rejecting connections. Because PostgreSQL uses one process per connection, every slot consumes memory and scheduler overhead. Once max_connections is reached, the server refuses new backends entirely. Raising max_connections without fixing the root cause increases memory pressure and context-switch thrashing. This guide covers how to distinguish a true capacity shortage from a leak or pool misconfiguration, how to recover safely, and how to prevent recurrence.

PostgreSQL connection refused: pg_hba, listen_addresses, and TCP diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL connection refused: pg_hba, listen_addresses, and TCP diagnosis

An application cannot reach PostgreSQL. The client reports either “Connection refused,” a hang until timeout, or “no pg_hba.conf entry.” These three symptoms point to different layers. Mixing them up leads to wasted restarts, overly broad firewall rules, or pg_hba.conf edits that never take effect. Work through the transport layer first, then the network path, then the authorization layer.

PostgreSQL defaults to binding only to the loopback interface. A fresh installation or container image rejects remote TCP attempts before pg_hba.conf is consulted. Changing listen_addresses requires a restart; changing pg_hba.conf only requires a reload. The diagnostic sequence is: confirm the listener, confirm the path, then confirm the rule.

PostgreSQL dead tuples piling up: why autovacuum can't keep up

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL dead tuples piling up: why autovacuum can’t keep up

Table sizes grow faster than insert rates, and queries that scanned thousands of rows last week now scan millions. pg_stat_user_tables shows n_dead_tup climbing into the millions while last_autovacuum is stale or absent. Autovacuum is running somewhere in the cluster, but it is losing the race.

PostgreSQL creates dead tuples on every UPDATE and DELETE. Autovacuum reclaims them only when the dead tuple count crosses a threshold. On large or high-churn tables, the default threshold is far too conservative, and even when vacuum does fire, long-running transactions, worker starvation, or streaming replica feedback can prevent tuple removal. The result is bloat: wasted space, slower scans, and eventually transaction ID wraparound risk.

PostgreSQL deadlock detected: how to diagnose and prevent deadlocks

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL deadlock detected: how to diagnose and prevent deadlocks

ERROR: deadlock detected means PostgreSQL aborted one transaction to break a circular wait-for graph. The victim returns SQLSTATE 40P01; the application must retry it. Deadlocks are a safety mechanism, not a bug: they fire when concurrent transactions acquire locks in incompatible orders. Even a few per minute degrade user experience, burn retry budget, and mask deeper contention. This guide shows how to read the deadlock output, find the root cause, and stop the cycle.

PostgreSQL disk full: emergency recovery and root cause analysis

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL disk full: emergency recovery and root cause analysis

When df -h shows 100% utilization on the PostgreSQL data volume, queries fail with “could not write to file”. If pg_wal fills, the server enters PANIC and refuses to restart until space is freed. The fastest way to make the incident worse is to delete WAL files from pg_wal manually. PostgreSQL needs those files for crash recovery; removing them causes data inconsistency that forces a restore from backup. Identify which subsystem is consuming space, reclaim it safely, and fix the root cause before the cycle repeats.

PostgreSQL ERROR: could not obtain lock — diagnosis and recovery

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL ERROR: could not obtain lock — diagnosis and recovery

ERROR: could not obtain lock on row in relation and ERROR: canceling statement due to lock timeout mean a query requested a lock but PostgreSQL refused to wait. The database is not down; a session is holding a resource another transaction needs.

Three variants produce these errors:

SELECT ... FOR UPDATE NOWAIT fails immediately if the row is locked.
A statement that exceeds lock_timeout fails after waiting.
DDL such as ALTER TABLE or CREATE INDEX requires AccessExclusiveLock and will wait or fail depending on session configuration.

These often cascade: one long-running query blocks a schema change, the schema change queues behind it, and subsequent queries queue behind the DDL until the connection pool exhausts.

PostgreSQL failover with Patroni: detection, promotion, and rollback

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL failover with Patroni: detection, promotion, and rollback

Patroni automates PostgreSQL failover with machine-enforced lease expiration. An agent on each node maintains a leader lock in a distributed consensus store such as etcd or Consul. When the primary stops renewing that lock, a surviving replica acquires it and promotes itself. Detection to promotion typically completes in seconds rather than the minutes a manual procedure requires.

Automation does not remove operational risk. A network blip between Patroni and its consensus store can look exactly like a primary death. A promoted replica with unbounded lag can wipe out your RPO. An old primary that restarts outside Patroni’s control can create a split brain. Understanding detection, promotion, and rollback mechanics is essential in production.

PostgreSQL FATAL: Too Many Connections - Causes & Fixes

Mon, 01 Jan 0001 00:00:00 +0000

Applications throw FATAL: sorry, too many clients already. Health checks fail. Retries amplify the problem. The database is not down, but it rejects new traffic.

PostgreSQL uses a process-per-connection model. Every backend holds memory and scheduler time even when idle. Raising max_connections usually deepens the problem because the root cause is why slots are occupied and what those backends are doing.

This guide shows how to diagnose connection exhaustion, distinguish PostgreSQL saturation from pooler exhaustion, and fix common root causes safely.

PostgreSQL frozen XID monitoring: catching wraparound 6 months early

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL frozen XID monitoring: catching wraparound 6 months early

Transaction ID wraparound is a slow burn, not a sudden crisis. The 32-bit XID counter advances with every transaction. If VACUUM freeze does not keep pace, PostgreSQL will eventually refuse new transactions to prevent data corruption. By the time built-in log warnings fire, you may have only hours of runway left. Monitor age(datfrozenxid) and age(relfrozenxid) with tiered thresholds so you act with months of lead time.

PostgreSQL fsync=off: why disabling it ruins durability

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL fsync=off: why disabling it ruins durability

You are reviewing a PostgreSQL instance that crashed during a kernel update and now fails to start with checksum or WAL errors. Or you are benchmarking write throughput and a guide suggests turning off fsync to remove the “fsync bottleneck.” Maybe you are auditing configuration drift and found fsync = off in a postgresql.conf that was copied from an old developer environment.

PostgreSQL’s default is fsync = on in all supported versions. Disabling it can make bulk loads and heavy write workloads appear dramatically faster because the database stops waiting for the operating system to flush write-ahead log (WAL) records to durable storage. The cost is that an operating system crash or power loss can leave the data directory in an unrecoverable state. WAL is the source of truth; fsync = off means the database trusts the OS page cache with that truth. When that trust is broken by a power event, the resulting corruption may not be detected until you attempt to start the server or query a specific table.

PostgreSQL idle in transaction: detecting and killing zombie sessions

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL idle in transaction: detecting and killing zombie sessions

When pg_stat_activity fills with idle in transaction sessions, autovacuum stalls, dead tuples accumulate, and applications throw “too many clients” errors while backends sit idle. These zombie sessions hold a transaction snapshot. Under MVCC, this prevents VACUUM from reclaiming dead tuples created after the transaction started, causing table bloat, blocked DDL, and transaction-ID wraparound pressure.

This guide shows how to detect these sessions, determine whether they are blocking work, terminate them safely, and prevent recurrence.

PostgreSQL index bloat: detection and REINDEX CONCURRENTLY recovery

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL index bloat: detection and REINDEX CONCURRENTLY recovery

Queries that used to return in milliseconds are now spiking to hundreds. Disk usage grows faster than insert volume. EXPLAIN shows a bitmap index scan pulling thousands of heap pages for a selective filter. The cause is usually index bloat: dead pages and fragmentation in B-tree indexes. Unlike table bloat, index bloat is not visible in pg_stat_user_tables, and it often degrades query latency or fills a volume before you notice it. Detect bloat with pgstattuple and recover online with REINDEX CONCURRENTLY.

PostgreSQL logical replication failures: conflicts, schema drift, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL logical replication failures: conflicts, schema drift, and recovery

Your subscriber is behind. pg_stat_subscription shows a stalled LSN, the apply worker is throwing errors, or worse: replication appears healthy while the subscriber silently diverges from the publisher. Logical replication does not replicate DDL, does not resolve row conflicts automatically, and will halt on the first integrity error it encounters. When it breaks, the failure is often on the subscriber, but the root cause may be schema drift on either side, a forgotten replication slot on the publisher, or a local write that violated the assumption that the subscriber is read-only.

PostgreSQL major version upgrade: pg_upgrade, logical replication, and rollback plans

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL major version upgrade: pg_upgrade, logical replication, and rollback plans

A major version upgrade is one of the highest-risk maintenance operations on a PostgreSQL cluster. Storage format, system catalogs, and the query planner all change between major versions, so you cannot simply restart the server with new binaries. Every major upgrade is a migration, even when it happens on the same host.

Most teams choose between two paths. pg_upgrade rewrites system catalogs while reusing or copying data files. It is fast but requires a downtime window. Logical replication streams row changes to a new cluster running the target version, enabling near-zero-downtime cutover but adding operational complexity. The wrong choice is usually the one made without understanding rollback boundaries.

PostgreSQL missing indexes: detection from pg_stat_statements and logs

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL missing indexes: detection from pg_stat_statements and logs

When the planner cannot find a suitable index path, it falls back to a sequential scan. On large tables, this turns millisecond queries into multi second outages.

PostgreSQL exposes evidence through built-in instrumentation: pg_stat_user_tables tracks sequential versus index scan ratios, pg_stat_statements surfaces the most time-consuming query fingerprints, and auto_explain logs execution plans that reveal Seq Scan nodes directly.

This guide gives an operator workflow to detect missing indexes using read-only checks and hypothetical index simulation. It assumes pg_stat_statements is enabled; if not, adding it to shared_preload_libraries requires a server restart.

PostgreSQL Monitoring Checklist: The Signals Every Production Database Needs

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL exposes hundreds of counters across the pg_stat_* views, yet most production outages trace back to a small set of undetected conditions. Bloat accumulates silently until vacuum cannot catch up. Replication lag grows until failover becomes a data-loss event. Transaction ID age crosses a threshold and the database stops accepting writes. The problem is rarely a lack of metrics. It is knowing which signals to instrument at each stage of operational maturity.

PostgreSQL monitoring maturity model: from reactive to self-healing

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL monitoring maturity model: from reactive to self-healing

Production PostgreSQL does not usually fail catastrophically; it drifts. An unwatched dashboard, an untested backup, a regressing query plan, or a filling replication slot slowly creates an incident. This model gives you eight observable stages to benchmark your operations, with measurable indicators and common stuck points from production runbooks. Use it to find your current stage, the next transition enabler, and the organizational traps that cause regression.

PostgreSQL out of memory: OOM killer, shared_buffers, and work_mem

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL out of memory: OOM killer, shared_buffers, and work_mem

Your PostgreSQL primary restarts without warning, or individual backends vanish from the process list. The kernel log shows Out of Memory: Killed process 12345 (postgres). Existing connections may survive, but new connections fail until the postmaster recovers. This is a memory accounting mismatch between Linux overcommit, PostgreSQL shared and private memory allocation, and how you size shared_buffers and work_mem.

Inside Kubernetes or containers, the symptom is identical but the mechanism differs: cgroup v2 memory.max triggers an immediate SIGKILL with no ENOMEM grace period. In both cases, stop guessing at memory limits and start budgeting.

PostgreSQL pg_upgrade failures: extension, collation, and locale gotchas

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL pg_upgrade failures: extension, collation, and locale gotchas

You ran pg_upgrade --check and it reported a collation version mismatch. Or the upgrade finished, your application reconnected, and PostGIS functions failed with could not access file '$libdir/postgis-3'. Maybe SELECT queries against text columns returned rows in the wrong order, or unique indexes threw violations against previously clean data. These are structural gaps between what pg_upgrade copies (catalog metadata) and what it does not verify (shared libraries, OS collation rules, and runtime configuration).

PostgreSQL pg_wal directory full: causes and emergency recovery

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL pg_wal directory full: causes and emergency recovery

Your paging system fires because the PostgreSQL primary has stopped accepting writes. The error log reports a disk-full condition, and the WAL volume is at 100%. You cannot simply delete files from pg_wal to free space: doing so corrupts the database and breaks replication.

PostgreSQL recycles WAL segments only after a checkpoint, and only when they are no longer needed for crash recovery, archiving, or replication slots. Archiving failures, stalled replication slots, or bulk loads that exceed max_wal_size cause unbounded accumulation. This guide covers identification, safe recovery, and prevention.

PostgreSQL prepared statement plan cache: generic vs custom plan pitfalls

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL prepared statement plan cache: generic vs custom plan pitfalls

When an application repeats the same query with different parameters, prepared statements avoid repeated parsing. But PostgreSQL’s plan cache does not work the way many operators expect. The server does not automatically cache the first plan it generates. It executes the first five invocations with custom plans that see the actual bound values. At the sixth execution, the planner compares the average cost of those custom plans against a generic plan built with placeholders. If the generic plan looks cheaper, the session switches to it permanently. There is no automatic reversion.

PostgreSQL replica disconnected: detecting and recovering streaming replication

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL replica disconnected: detecting and recovering streaming replication

When replay_lag grows and the primary’s pg_stat_replication no longer lists the standby, queries return stale data and failover is unsafe. PostgreSQL’s WAL receiver automatically retries when streaming breaks, cycling through WAL archive, local pg_wal, and streaming connections. That loop can mask the root cause while the replica drifts toward an unrecoverable gap. Determine whether the replica is temporarily stalled, beyond recovery, or missing its replication slot on the primary.

PostgreSQL replica out of sync: timeline mismatches and recovery

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL replica out of sync: timeline mismatches and recovery

A PostgreSQL streaming replica that was healthy yesterday now refuses to start with a timeline mismatch error, or a former primary that you brought back online cannot rejoin the cluster as a standby. The log shows requested timeline N is not a child of this server's history and the replica loops in crash recovery while the current primary continues to diverge. This guide covers identifying the divergence point, choosing between pg_rewind and a full re-clone, and recovering the replica without introducing split-brain or data loss.

PostgreSQL replication lag: detection, diagnosis, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL replication lag: detection, diagnosis, and fixes

Replication lag is the distance between the last WAL record generated on the primary and the last record applied on a replica. In asynchronous streaming replication, a few seconds of lag is normal. When lag grows without bound, your recovery point objective becomes fiction.

Lag often grows silently. Replication processes stay connected, WAL streams flow, and uptime checks stay green while the byte gap creeps from megabytes to gigabytes. Promoting a replica that is hours behind destroys the consistency your application assumes.

PostgreSQL replication slot bloat: when a stale slot fills the disk

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL replication slot bloat: when a stale slot fills the disk

Disk is filling on the primary. Table sizes are stable and active replicas show healthy replication lag, but WAL in pg_wal keeps growing and is not being recycled. The most likely cause is a stale replication slot. When a logical subscriber, CDC connector, or physical replica disconnects without cleaning up its slot, PostgreSQL retains every WAL segment from the slot’s restart_lsn onward. This retention ignores max_wal_size unless max_slot_wal_keep_size is set to a finite value. The default is -1 (unlimited). The primary will retain WAL until the disk fills and writes halt. This guide covers confirmation, recovery, and prevention.

PostgreSQL row-level lock contention: SELECT FOR UPDATE patterns and fixes

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL row-level lock contention: SELECT FOR UPDATE patterns and fixes

P99 latency doubles. pg_stat_activity shows wait_event_type = 'Lock'. Queries are simple, indexes are present, and CPU is idle. The culprit is usually a row-level lock queue: one transaction holds a tuple lock longer than expected, and every subsequent transaction touching the same row waits in FIFO order.

This is not a deadlock. PostgreSQL detects deadlocks automatically and kills one participant. Lock contention is different: a slow or abandoned transaction acts as a head-of-line blocker. Unless you are polling pg_locks and transaction age, the queue is invisible in query logs. The pattern is most common with SELECT ... FOR UPDATE in database-backed job queues, UPDATE statements that silently take stronger locks than intended, and session-level advisory locks that survive rollback.

PostgreSQL sequential scan on a large table: when to add an index and when not to

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL sequential scan on a large table: when to add an index and when not to

Seeing Seq Scan on a large table in EXPLAIN does not mean the planner is wrong. Adding a B-tree index often makes the query slower, burdens every write path with index maintenance, and wastes storage. PostgreSQL’s planner compares the estimated cost of reading the table sequentially against the estimated cost of traversing an index and fetching rows from the heap. There is no hardcoded row-percentage threshold. On SSD-backed servers, the default cost parameters are frequently stale, so the planner may be wrong in either direction.

PostgreSQL shared_buffers tuning: 25% of RAM and why that rule of thumb breaks

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL shared_buffers tuning: 25% of RAM and why that rule of thumb breaks

The PostgreSQL documentation suggests 25% of RAM as a starting value for shared_buffers on dedicated servers and warns that values above 40% rarely improve performance. In production, operators routinely use 25% as a default and then encounter checkpoint storms, OOM kills, or degraded analytical query performance. The rule breaks because PostgreSQL does not coordinate with the Linux kernel page cache. The same page can exist in both shared_buffers and the OS page cache, so an oversized PostgreSQL cache starves the kernel of memory needed for sequential scans, WAL, and temporary files.

PostgreSQL Slow Queries: Diagnosis From Log To Plan To Fix

Mon, 01 Jan 0001 00:00:00 +0000

A query that returned in 10 ms yesterday is now taking 8 seconds. No deploys, no schema changes. Before adding an index or restarting the database, determine whether the slowness is in the plan, the data, or the environment. This guide covers a three-layer workflow: log-based discovery with log_min_duration_statement, aggregate profiling with pg_stat_statements, and per-query execution plan capture with auto_explain and manual EXPLAIN (ANALYZE, BUFFERS).

flowchart TD
 A[Slow query reported] --> B[Check pg_stat_statements for mean_time and stddev_time]
 B --> C{High stddev relative to mean?}
 C -->|Yes| D[Plan flapping or parameter skew]
 C -->|No| E[Stable plan or system bottleneck]
 D --> F[Capture plan with auto_explain or EXPLAIN]
 E --> F
 F --> G{Estimated rows far from actual?}
 G -->|Yes| H[Stale statistics or skewed data]
 G -->|No| I[Bloat, locks, or cache pressure]

What This Means

A slow query is a symptom. PostgreSQL’s planner chooses a path based on statistics from ANALYZE, bound parameter values, and configuration such as work_mem. When these inputs change, the same query text can switch from a hash join to a nested loop and slow down by orders of magnitude.

PostgreSQL split-brain after failover: detection and reconciliation

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL split-brain after failover: detection and reconciliation

A failover should leave one primary and a clean topology. Split-brain means two instances accept writes, application connections are split across both nodes, and transaction histories diverge. It typically starts with a network partition, a missed demotion signal, or a health-check false positive that promotes a replica while the old primary keeps running. Once both primaries accept transactions, timelines diverge and you must reconcile.

PostgreSQL statistics out of date: ANALYZE, default_statistics_target, and bad plans

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL statistics out of date: ANALYZE, default_statistics_target, and bad plans

When EXPLAIN shows a nested loop joining a million-row table, or a sequential scan where an index should win, and the query text has not changed, stale planner statistics are the likely culprit.

PostgreSQL’s cost-based optimizer relies on ANALYZE-derived catalog data to estimate cardinality. ANALYZE samples table contents and writes histograms, most-common-value lists, and correlation data into pg_statistic. When that data no longer matches the on-disk distribution, the planner misestimates row counts. A cardinality underestimate favors a nested loop; an overestimate favors a sequential scan. The result is a sudden latency spike that connection pooling or hardware scaling will not fix.

PostgreSQL streaming replication broken: how to rebuild without full basebackup

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL streaming replication broken: how to rebuild without full basebackup

When a replica logs requested WAL segment has already been removed, or after failover when the old primary must rejoin as a standby, a full pg_basebackup on a multi-terabyte database can take hours and saturate network and disk. If data files have diverged but are mostly identical, pg_rewind can resync by copying only changed blocks. It is faster than a base backup, but has strict prerequisites. If pg_rewind crashes mid-operation, the target data directory is likely corrupt and only a fresh base backup is safe.

PostgreSQL synchronous_commit: durability vs throughput trade-offs

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL synchronous_commit: durability vs throughput trade-offs

synchronous_commit moves the durability boundary between a client receiving COMMIT OK and the data actually surviving a crash. The default, on, flushes local WAL to disk. With synchronous replication enabled, it also waits for a standby to flush. The five values (off, local, on, remote_write, and remote_apply) trade latency against survival guarantees. The wrong choice either accepts unplanned data loss or strangles write throughput with network round-trips.

PostgreSQL table bloat: detection, measurement, and remediation

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL table bloat: detection, measurement, and remediation

Table bloat appears as tables growing while row counts stay flat, sequential scans slowing despite indexes, and disk alarms that do not track business growth. PostgreSQL’s MVCC writes new tuple versions instead of overwriting old ones; every UPDATE leaves a dead row and every DELETE leaves invisible garbage. VACUUM reclaims that space for reuse within the file, but plain VACUUM does not shrink the relation on disk. When dead tuples outpace cleanup, a table can become several times larger than its logical content.

PostgreSQL tablespace management: moving data without downtime

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL tablespace management: moving data without downtime

PostgreSQL tablespaces map relations to specific filesystem paths. Moving data between tablespaces is not an online operation in core PostgreSQL: ALTER TABLE SET TABLESPACE copies the relation’s data files while holding an ACCESS EXCLUSIVE lock for the entire duration. For a large relation, reads and writes block for minutes or hours. This guide covers three operational paths: the built-in offline move for small or maintenance-tolerant relations, offline symlink relocation for entire tablespace directories, and near-zero-downtime workarounds for heavy tables.

PostgreSQL temp file explosion: work_mem tuning and disk pressure

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL temp file explosion: work_mem tuning and disk pressure

Disk usage on your PostgreSQL primary is climbing by gigabytes per hour. Query latency has doubled, iowait is spiking, and you have not deployed new code. The database is writing large temporary files to disk. In-memory operations such as sorts, hash joins, and materialization are spilling because they exceed the per-operation memory budget. The root cause is almost always a mismatch between work_mem and actual query memory demand, compounded by the fact that PostgreSQL applies this limit per plan node, not per query or session.

PostgreSQL TOAST table bloat: detection and recovery for large columns

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL TOAST table bloat: detection and recovery for large columns

Unexplained disk growth on a PostgreSQL instance often traces to TOAST bloat. A table with large JSONB, text, or bytea columns grows far beyond live data size, and queries that materialize those columns slow down. A plain VACUUM does not shrink the on-disk footprint because the dead tuples are in the companion pg_toast table, not the main heap.

TOAST (The Oversized-Attribute Storage Technique) moves values exceeding the TOAST_TUPLE_TARGET out of the main row and into a companion pg_toast.NNNN table. PostgreSQL compresses the value and splits it into roughly 2 KB chunks, each stored as a separate row with a unique index on (chunk_id, chunk_seq). Updates or deletes to the row leave dead chunks in the TOAST table. Because operators often monitor only the parent table, TOAST bloat grows silently until it dominates disk usage.

PostgreSQL transaction ID wraparound: detection and emergency recovery

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL transaction ID wraparound: detection and emergency recovery

PostgreSQL can suddenly stop accepting writes and emit warnings that the database must be vacuumed within a shrinking number of transactions. This is transaction ID wraparound. It is not gradual performance degradation; it is a hard stop that can take a database offline for hours if old tuples are not frozen in time.

Every write transaction consumes a 32-bit XID. After roughly two billion transactions, the counter nears the point where older tuples could appear to belong to the future, corrupting visibility. PostgreSQL refuses to hand out new XIDs once fewer than roughly three million remain. Warnings appear at roughly forty million remaining. The defense is VACUUM freeze, which marks old rows with FrozenTransactionId so they no longer depend on the live counter.

PostgreSQL VACUUM FULL vs pg_repack: choosing the right bloat fix

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL VACUUM FULL vs pg_repack: choosing the right bloat fix

Table bloat is the gap between logical data size and on-disk consumption. Heavy UPDATE and DELETE activity leaves dead tuples in the heap. Autovacuum reclaims those tuples for reuse but does not shrink the underlying file. Once sequential scans slow and disk space vanishes, operators must choose between VACUUM FULL and pg_repack. The wrong choice locks a table for hours, exhausts disk mid-operation, or leaves orphaned triggers behind. This guide explains how each tool rewrites the heap, what locks it takes, and how to pick the right one.

PostgreSQL WAL archive failures: archive_command exit codes and recovery

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL WAL archive failures: archive_command exit codes and recovery

WAL archiving is the durability bridge between your PostgreSQL primary and your ability to recover to any point in time. When archive_command fails, WAL segments stay pinned in pg_wal/. The directory grows until the filesystem fills, at which point PostgreSQL performs an emergency PANIC shutdown. Even before that happens, every failed segment is a gap in your backup chain, rendering base backups useless for PITR beyond the first missing file.

PostgreSQL: checkpoints are occurring too frequently -- what to tune

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL: checkpoints are occurring too frequently – what to tune

Your PostgreSQL logs show LOG: checkpoints are occurring too frequently (9 seconds apart) with the hint Consider increasing the configuration parameter 'max_wal_size'. Sustained I/O latency spikes correlate with WAL segment rotation. On write-heavy primaries, this almost always means max_wal_size is too small for the workload.

A checkpoint flushes all dirty shared buffers to disk. By default, a checkpoint fires every 5 minutes (checkpoint_timeout) or every 1 GB of WAL (max_wal_size), whichever comes first. On a busy OLTP primary, 1 GB of WAL can accumulate in minutes. When max_wal_size triggers the checkpoint, you get a forced (requested) checkpoint. These are unpredictable, collide with existing write load, and cause visible latency spikes.

PostgreSQL: database is not accepting commands to avoid wraparound data loss

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL: database is not accepting commands to avoid wraparound data loss

Writes fail abruptly. SELECT still works, but INSERT, UPDATE, DELETE, and DDL return an error like:

ERROR: database is not accepting commands that assign new XIDs to avoid wraparound data loss in database "..."

This is PostgreSQL’s emergency brake, not a crash. The engine has stopped issuing new transaction IDs to prevent tuple visibility corruption. Modern PostgreSQL lets you recover without restarting or entering single-user mode. Remove whatever is blocking VACUUM progress, then freeze old rows.

Reading EXPLAIN ANALYZE: the operator's guide to PostgreSQL query plans

Mon, 01 Jan 0001 00:00:00 +0000

Reading EXPLAIN ANALYZE: the operator’s guide to PostgreSQL query plans

When pg_stat_statements flags a query as a top consumer, EXPLAIN ANALYZE is the operator’s ground truth. It shows what the executor did, node by node, buffer by buffer. Misreading the output leads to useless indexes and production changes that make performance worse.

This guide covers the mechanics that matter in production: how actual time accumulates through the node tree, why estimated rows diverge from reality, when buffer counts reveal cache misses versus disk reads, and how artifacts like the loops multiplier hide expensive nodes. It is a field manual for deciding, in the next five minutes, whether the problem is a missing index, stale statistics, a bad plan choice, or something deeper.