vCenter vPostgres ’too many clients already’: connection exhaustion

vCenter operations time out. Power-on tasks hang in “Running” state. The vSphere Client is sluggish or fails to load. PowerCLI sessions return errors. Backup jobs fail mid-run. SSH into the VCSA and tail the vpxd or vPostgres logs, and you find the string that anchors the incident: FATAL: sorry, too many clients already.

The embedded vPostgres database is rejecting new connections because it has reached max_connections. VMware tunes this value per deployment size, and a tuning script rewrites it based on the appliance’s memory allocation. At the limit, anything needing a database connection fails immediately. This is a cliff-edge failure, not graceful degradation.

The fix is not to raise max_connections. Find the client generating the sessions. The database is rarely the root cause.

What this means

vpxd, the core vCenter daemon, connects to vPostgres for every inventory read, task state write, event insert, and statistics rollup. Other VCSA services also open database connections. The total concurrent count is bounded by max_connections, which VMware sets based on VCSA memory allocation. When active connections hit the ceiling, the postmaster rejects new attempts with FATAL: sorry, too many clients already.

The rejection is binary. A service that needs a connection either gets one or does not. There is no queueing at the database layer. vpxd task processing stalls because it cannot persist task state. Stats rollup falls behind because it cannot write. Host heartbeat processing slows because inventory updates cannot be committed. The visible symptom is “vCenter is slow or broken,” but the mechanism is a saturated connection pool behind a fixed PostgreSQL limit.

The correlate that almost always appears alongside this error is a growing vpxd task queue. Tasks pile up in “Running” or “Queued” state because the threads that would execute them are blocked acquiring database connections. If you see the error string and task queue depth climbing simultaneously, you are looking at the same failure from two angles.

flowchart TD
    A[SDK clients open sessions] --> B[vpxd executes tasks]
    B --> C[Each task needs vPostgres connection]
    C --> D{Active connections below max?}
    D -- yes --> E[Task proceeds]
    D -- no --> F[FATAL: too many clients already]
    F --> G[Task stalls in queue]
    G --> H[vpxd CPU rises, SDK response degrades]
    H --> B

Peak connection count should stay below roughly 70% of max_connections. Above that, operational bursts, maintenance connections, and superuser reserved connections leave no headroom. Once you cross the line, the cliff is immediate.

Common causes

CauseWhat it looks likeFirst thing to check
Backup solution stormDozens or hundreds of snapshot create/delete operations in a narrow window, all from one client IPActive SDK sessions by client IP, backup job state
Monitoring or automation tool hammering the APIA single session UUID making thousands of calls to PerformanceManager.queryAvailableMetric, vPostgres CPU near 100%vpxd-profiler.log for the offending session UUID and ClientIP
PowerCLI or custom script opening many concurrent SDK sessionsConnection count ramps in step with script execution, no single long-running querySDK session count by client IP, script source
DRS aggressiveness too high in a churn-heavy clusterMigration rate elevated, vMotion tasks dominate the queue, connections rise with migration burstsDRS migration events, aggressiveness level
Stats rollup overlapping with heavy operational window5-minute CPU spikes on vpxd coincide with task saturation, rollup falling behindStats rollup lag, statistics level configuration
VCHA passive node undersizedReplication errors such as “PostgreSQL replication is not in progress”, passive node memory lower than activePassive node memory vs active, pg_stat_replication
Storage latency on the VCSA VM’s datastoreTransaction delays hold connections longer, compounding exhaustionVCSA datastore latency (DAVG/KAVG/GAVG)

Quick checks

These are read-only and safe to run during an incident. Run psql as the postgres OS user (su - postgres) to avoid peer auth issues. If psql fails with a socket path error, set PGHOST=/var/run/vpostgres explicitly; without it, psql looks in /tmp/ and fails.

# Current max_connections as tuned by the appliance
/opt/vmware/vpostgres/current/bin/psql -U postgres -A -t -c "SHOW max_connections;"

# Active connection count and ratio against max
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT count(*) AS active_connections, (SELECT setting::int FROM pg_settings WHERE name='max_connections') AS max_connections FROM pg_stat_activity;"

# Long-running queries holding connections
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT pid, now() - query_start AS duration, state, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC LIMIT 10;"

# Long-running transactions that block autovacuum and hold the horizon
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT pid, now() - xact_start AS xact_duration, state, query FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY xact_start LIMIT 10;"

# VCSA service health
service-control --status --all

# vpxd errors related to database connectivity
grep -i "database.*connect\|postgres.*fail\|ODBC\|too many clients" /var/log/vmware/vpxd/vpxd.log | tail -50

# Offending SDK session if a monitoring tool is the cause
grep -i "queryAvailableMetric" /var/log/vmware/vpxd/vpxd-profiler.log | tail -20

# Database partition usage (full disk produces a different but related failure)
df -h /storage/db

# VCHA replication state, if configured
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT client_addr, state, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes FROM pg_stat_replication;"

How to diagnose

  1. Confirm the error is vPostgres-side. Grep the vpxd log for the exact string and for ODBC or database connectivity errors. If the string appears, you are in connection exhaustion.
  2. Read max_connections and the active count from pg_stat_activity. Compute the ratio. Above 70% of max at peak is a headroom problem. At 100%, you are in the incident.
  3. Identify what is holding connections. Sort pg_stat_activity by duration and by application name if available. Long-running queries suggest stats rollup or a misbehaving query. Many short-lived sessions from the same client suggest an SDK storm.
  4. Cross-reference with the vpxd task queue. A queue growing while vpxd CPU is elevated and connections are near max confirms the overload cascade.
  5. Identify the offending SDK client. Check SDK session count by client IP. Check vpxd-profiler.log for repeated calls to PerformanceManager.queryAvailableMetric from a single session UUID and ClientIP. That is the classic signature of a monitoring or automation tool generating sessions.
  6. Rule out storage latency on the VCSA VM’s datastore. High DAVG or KAVG stretches transaction times, which holds connections longer and worsens exhaustion. The VCSA VM should not sit on an overcommitted or slow datastore.
  7. If VCHA is configured, verify the passive node has memory identical to the active node. A passive node with less memory gets a lower max_connections from the tuning script, and the active node cannot open enough connections for PostgreSQL replication, producing “PostgreSQL replication is not in progress” errors that share the root cause.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Active vPostgres connections vs max_connectionsThe direct ratio that tells you how close you are to the cliffPeak sustained above 70% of max
vpxd task queue depthTasks stall when vpxd cannot get database connectionsQueue depth sustained above 0 during non-peak hours, or growing
vpxd CPU utilizationCompute-bound vpxd cannot process tasks even when connections free upSustained above 70% of allocated cores
SDK session count by client IPPinpoints the integration driving connection growthSingle client IP holding a disproportionate session count
queryAvailableMetric call rate in vpxd-profiler.logClassic signature of a monitoring tool hammering the APISingle session UUID making thousands of calls
vPostgres longest-running transactionLong transactions hold connections and block autovacuumAny transaction open longer than 1 hour
VCSA datastore latency (GAVG/DAVG/KAVG)Storage latency lengthens transactions, compounding exhaustionGAVG sustained above 30ms, or DAVG rising on the VCSA VM’s datastore
VCHA replication lag and passive node memoryUndersized passive node breaks replication with the same root causeReplication lag growing, passive node memory below active
/storage/db utilizationDisk exhaustion produces a different failure but often co-occursAbove 80%, or WAL directory growing

Fixes

Do not raise max_connections by hand. It is tuned to the appliance’s resources, and the tuning script overwrites it on the next vmware-vpostgres restart. Treat the database limit as a fixed boundary and fix the load.

Throttle or disconnect the offending SDK client

If SDK session counts by client IP point to a single integration, that is the root cause. Throttle the client, pause the backup job, or disconnect the rogue session. For monitoring tools hammering PerformanceManager.queryAvailableMetric, the fix lives in the tool’s configuration: reduce collection frequency, reduce the metric scope, or upgrade the tool if a vendor fix exists.

Reduce the vpxd task load

If the queue is growing because of legitimate operational burst, reduce concurrency. For DRS storms, temporarily set DRS to manual. For post-maintenance bursts, stagger hosts exiting maintenance mode. For backup storms, spread backup jobs across wider windows so snapshot create/delete operations do not pile up.

Address stats rollup overlap

If stats rollup overlaps with heavy operational windows and consumes connections, check the statistics level. Levels 3 and 4 generate substantially more write load than levels 1 and 2. Reducing to level 1 or 2 lowers baseline connection pressure from rollup.

Clear long-running transactions

Long-running idle-in-transaction sessions hold connections and block autovacuum across all tables. Identify them with the long-running transaction query and work with the owning service or integration to close them. Do not kill PostgreSQL backends casually during an incident without understanding what vpxd or another VCSA service is doing.

Fix storage latency on the VCSA datastore

If DAVG or KAVG on the VCSA VM’s datastore is elevated, transactions take longer, connections are held longer, and the exhaustion cliff arrives sooner. Migrate the VCSA VM to faster or less contended storage. The VCSA should not sit on an overcommitted host or a noisy datastore.

Correct VCHA passive node sizing

If the passive node has less memory than the active, the tuning script assigns it a lower max_connections, replication breaks, and the active node accumulates WAL. Size the passive node identically to the active. Restarting vmware-vpostgres via vmon-cli --restart vmware-vpostgres reruns the tuning logic and applies the corrected value.

When restarting is unavoidable

Restarting vpxd drops all in-flight tasks and requires minutes to tens of minutes to rebuild the inventory cache from the database. Use it only when the offending client cannot be throttled and the queue is irrecoverable. Restarting vmware-vpostgres will not raise max_connections above what the appliance memory supports, so it is not a workaround for exhaustion.

Prevention

  • Track active connections against max_connections continuously. Alert at 70% sustained. Treat 90% as a page.
  • Monitor SDK session count by client IP. Baseline each integration. A new integration or a misconfigured collection interval is the most common recurring cause.
  • Monitor vpxd task queue depth and average task duration. Sustained queue depth above 0 outside peak hours is an early signal.
  • Keep statistics level at 1 or 2 unless a specific investigation requires more. Return to baseline after the investigation.
  • Keep the VCSA VM on fast, dedicated storage. Monitor its datastore latency independently of the rest of the cluster.
  • In VCHA configurations, size the passive node identically to the active. Monitor replication lag.
  • Track vpxd log error rate. The “too many clients already” string is the terminal symptom. Earlier signals appear as ODBC warnings and database connectivity errors.
  • Re-baseline after every new integration. Backup tools, monitoring platforms, orchestration, and custom scripts are each potential session sources.

How Netdata helps

  • Per-second PostgreSQL connection metrics show the ramp toward max_connections before the error string appears in logs.
  • Correlating active connection count with vpxd CPU and task queue depth distinguishes a connection-bound stall from a compute-bound stall, which changes the fix.
  • Per-client SDK session tracking requires custom instrumentation (log parsing of vpxd-profiler.log), but once collected, it surfaces the misbehaving integration by IP so you can throttle the right client instead of restarting services.
  • Anomaly detection on connection count catches the monitoring-tool-hammering pattern early, before vPostgres CPU pins at 100%.
  • VCSA VM-level metrics from the hypervisor (CPU ready, datastore latency, memory balloon) reveal whether storage or host contention is lengthening transactions and compounding the exhaustion.