vCenter /storage/db full: vPostgres stops and the whole management plane dies

The vSphere Client returns 503 Service Unavailable. PowerCLI sessions hang and time out. DRS has stopped evaluating, vMotion orchestration is gone, and provisioning fails. Running VMs on the ESXi hosts continue to operate, but the management plane is gone.

The root cause is almost certainly the /storage/db partition on the vCenter Server Appliance (VCSA). This is where vPostgres keeps its data files. At 95% utilization on any partition, VMware automatically shuts down vmware-vpxd to protect the database from corruption. At 100%, vPostgres cannot extend a data file or write a WAL record and crashes. Once vPostgres is down, vpxd has no database and cannot restart.

The trap: root / can look healthy while /storage/db is at 100%. The VCSA uses dedicated partitions that fill independently. Standard Linux disk monitoring that only checks root misses this failure entirely.

This is a cliff-edge failure with no graceful degradation. Recovery requires freeing space on /storage/db without corrupting the database, which means following a specific sequence and avoiding several destructive shortcuts.

What this means

PostgreSQL does not tolerate ENOSPC on its data directory. When /storage/db fills, any operation that requires extending a data file, writing a WAL record, creating a temporary sort file, or running a vacuum fails. The vPostgres log captures this as could not extend file ... No space left on device. The vpxd log records No Space left on the device. vPostgres crashes, and because vpxd depends on the database for every operation, the management plane goes dark.

The 95% auto-shutdown is a VMware safety mechanism, not a crash. When any partition reaches 95%, vmware-vpxd is killed automatically to prevent database corruption. The vSphere Client returns 503 errors. If you catch the problem here, recovery is simpler because vPostgres is still running and the partition has not hit 100%.

At 100%, vPostgres itself stops. vpxd cannot start without its database. DRS, vMotion, provisioning, alarm management, and statistics collection all cease. HA continues independently through the FDM agents on ESXi hosts, so VMs can still be restarted after host failure, but you have lost all management visibility and orchestration.

flowchart TD
    A["/storage/db reaches 95%"] --> B["vpxd auto-shutdown"]
    B --> C["vSphere Client 503
DRS stops, provisioning lost"]
    A --> D["/storage/db reaches 100%"]
    D --> E["vPostgres cannot write WAL
or extend data files"]
    E --> F["vPostgres crashes"]
    F --> G["vpxd has no database"]
    G --> H["Management plane down
VMs still running on hosts"]

Common causes

CauseWhat it looks likeFirst thing to check
SEAT table bloat (stats, events, alarms, tasks)/storage/db grows steadily; vpx_event, vpx_task, or vpxd_hist_stat* are the largest tablesRun the largest-tables query in Quick checks
Failed purge jobPurge runs as a vpxd internal task; if vpxd was overloaded or down, old data was never removedCheck vpx_parameter for retention settings; compare vpx_event row count to retention window
Statistics level too high (Level 3 or 4)vpxd_hist_stat* tables dominate database size; growth accelerates after a level changeSELECT * FROM vc.vpx_parameter WHERE name LIKE '%stats%';
WAL accumulation from VCHA replication lagActive node WAL directory grows; passive node is behind or unreachableCheck pg_stat_replication for replay lag bytes
WAL archive mode without archive_command/storage/dblog fills on 7.0+; WAL directory on /storage/db grows on 6.xCheck archive_mode and archive_command in postgresql.conf

Version note: in vCenter 6.5/6.7, the WAL directory (pg_xlog) lives under /storage/db/vpostgres/. In vCenter 7.0+, WAL moved to /storage/dblog/vpostgres/pg_wal. WAL accumulation in 7.0+ fills /storage/dblog, not /storage/db. Data file bloat fills /storage/db in all versions. Also note: /storage/archive at 100% is normal by design in vCenter 6.7 and later and can be safely ignored.

Quick checks

All commands are read-only and safe. If vPostgres has crashed, the psql commands will fail with a connection error, which itself confirms the diagnosis.

# Check all partitions - root may be fine while /storage/db is at 100%
df -h

# Check inode usage separately - small files can exhaust inodes before space
df -i

# Check service health across all VCSA services
service-control --status --all

# Check the two services that matter most here
service-control --status vpxd
service-control --status vmware-vpostgres

# Total database size
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c \
  "SELECT pg_size_pretty(pg_database_size('VCDB'));"

# Top 20 largest tables - identifies SEAT bloat
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c \
  "SELECT schemaname, tablename, \
   pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as total_size \
   FROM pg_tables \
   WHERE schemaname NOT IN ('pg_catalog', 'information_schema') \
   ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC LIMIT 20;"

# WAL directory size - path differs by version
du -sh /storage/db/vpostgres/pg_xlog/      # vCenter 6.5/6.7
du -sh /storage/dblog/vpostgres/pg_wal/    # vCenter 7.0+

# Statistics level and retention settings
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c \
  "SELECT * FROM vc.vpx_parameter WHERE name LIKE '%stats%';"

How to diagnose it

  1. Confirm which partition is full. Run df -h and look specifically at /storage/db. Do not trust the VAMI UI, which rounds aggressively. Root / may show ample free space while /storage/db is at 100%.

  2. Check whether vpxd is running. Run service-control --status vpxd. If it is stopped and /storage/db is above 95%, the auto-shutdown mechanism triggered. If /storage/db is at 100%, vPostgres has also crashed.

  3. Check vPostgres status. Run service-control --status vmware-vpostgres. If it is stopped, check /var/log/vmware/vpostgres/postgresql.log for No space left on device errors confirming the crash cause.

  4. Identify what is consuming the space. If vPostgres is still running, connect and run the largest-tables query from Quick checks. The vpx_event, vpx_task, vpx_event_arg, and vpxd_hist_stat* tables are the usual culprits.

  5. Check dead tuple ratio. High dead tuple counts mean autovacuum is falling behind and tables are bloated with unreclaimed space.

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c \
  "SELECT schemaname, relname, n_live_tup, n_dead_tup, \
   round(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) as dead_pct \
   FROM pg_stat_user_tables WHERE n_dead_tup > 10000 \
   ORDER BY n_dead_tup DESC LIMIT 20;"
  1. Check VCHA replication lag if applicable. WAL accumulates on the active node when the passive node cannot replay fast enough.
# vCenter 7.0+ (PostgreSQL 10+)
/opt/vmware/vpostgres/current/bin/psql -U postgres -c \
  "SELECT client_addr, state, \
   pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) as replay_lag_bytes \
   FROM pg_stat_replication;"
# vCenter 6.5/6.7 uses pg_xlog_location_diff instead of pg_wal_lsn_diff
<!-- TODO: verify exact function name and column names for 6.x VCHA replication query -->

  1. Check statistics level. Level 3 or 4 generates dramatically more data than Level 1 or 2. If someone elevated the level for troubleshooting and forgot to lower it, that is likely the root cause of sustained growth.

  2. Check for crash loop behavior. If vmon has restarted vpxd multiple times, it may have given up. The service stays stopped with no automatic recovery. This is silent unless you monitor restart counts.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
/storage/db partition utilizationvPostgres crashes at 100%; vpxd auto-shuts at 95%Trending above 70%; sustained growth
vPostgres service statusvpxd cannot function without itSTOPPED or FAILED state
vpxd service statusManagement plane is down without itSTOPPED after VCSA uptime exceeds 600 seconds
Database total size and growth ratePredicts when the partition will exhaustGrowth exceeding 1 GB/day without inventory growth
Dead tuple ratio on major tablesIndicates autovacuum is falling behindAbove 30% on vpx_event or vpxd_hist_stat*
WAL directory sizeWAL accumulation signals replication or checkpoint issuesSustained growth, especially with VCHA
VCHA replication lagLag causes WAL accumulation on active nodeReplay lag above 100 MB or growing
Statistics level settingLevel 3/4 generates excessive dataAnything above Level 2 without explicit justification
/storage/log partition utilizationLog bombs can trigger the same death spiralAbove 85%, or rapid growth during incidents

Fixes

Free space immediately (triage)

If /storage/db is at 100% and vPostgres has crashed, you need space before anything else.

Do not manually delete WAL files from the WAL directory. This can corrupt the database and require a full vCenter restore.

Safe immediate actions:

  • Truncate large log files on /storage/log, not /storage/db. Use > /path/to/logfile to truncate in place, not rm. This frees space on the log partition and may help if vPostgres was indirectly affected by log partition pressure.
  • Remove old core dumps from /storage/core if present. These can be gigabytes each from prior vpxd crashes.

If /storage/db itself is the problem and the above does not help, you need to expand the partition before running cleanup queries. vPostgres needs working space for VACUUM and sort operations, and at 100% there is none.

Reduce statistics level

If statistics level is 3 or 4, lowering it to 1 or 2 stops the bleeding. This does not shrink existing data but prevents future growth from the same cause. This is the most common configuration mistake behind slow-burn database bloat.

Address SEAT table bloat

If vpx_event, vpx_task, or vpxd_hist_stat* tables are the largest, retention may be too long or the purge job failed. The purge runs as a vpxd internal task; if vpxd was overloaded or down, old data accumulated.

After correcting retention settings, PostgreSQL reclaims space within tables for reuse but does not return it to the filesystem without VACUUM FULL. VACUUM FULL locks the table for its duration and requires free space equivalent to the table being vacuumed. This creates a chicken-and-egg problem when the partition is full.

Tradeoff: You may need to expand the partition first to get working space, run VACUUM FULL on the offending tables during a maintenance window, then optionally shrink the partition back. Do not tune autovacuum settings manually without VMware support guidance.

Fix VCHA replication lag

If the passive node is behind, WAL accumulates on the active node. Check the VCHA network for packet loss, MTU mismatch, or bandwidth constraints on the dedicated replication NIC.

Fix WAL archive mode (if applicable)

If archive_mode is enabled but archive_command is empty or unset, PostgreSQL retains WAL files indefinitely. The fix is to set archive_mode = off in /storage/db/vpostgres/postgresql.conf. This directly affects /storage/dblog in 7.0+ but prevents a cascade that can destabilize vPostgres overall.

Expand the partition

The VAMI interface on port 5480 provides storage expansion. This increases the VCSA virtual disk and expands the logical volumes, including /storage/db. This is the cleanest path when the database legitimately needs more space for its working set or when you need room to run VACUUM FULL.

Watch for the log rotation regression

If you recently upgraded to vCenter 8.0U3g and /storage/log is filling, check whether postgresql.log in /var/log/vmware/vpostgres/ has stopped rotating and grown to tens of gigabytes. The fix involves correcting the logrotate entry and zeroing out the existing file. While this fills /storage/log rather than /storage/db, it can trigger the same disk space death spiral where service crashes generate more logs, which fills the partition further.

Prevention

  • Monitor /storage/db specifically. Do not monitor only root /. The VCSA partition layout means partitions fill independently and root can be healthy while /storage/db is at 100%.
  • Keep at least 30% free on /storage/db. vPostgres needs working space for WAL, temporary files, sort operations, and vacuum. The 95% auto-shutdown leaves no margin.
  • Alert at 70% and 85%. Give yourself time before the 95% mechanism fires. The 85% threshold is the point of immediate action.
  • Track database growth rate. Divide remaining free space by daily growth rate to estimate runway. Growth exceeding 1 GB/day in a stable environment is a red flag.
  • Audit statistics level after any troubleshooting. Level 3 or 4 left in production is one of the most common causes of database bloat.
  • Monitor dead tuple ratio on major tables. Above 20% means autovacuum is falling behind and tables are bloating.
  • Monitor VCHA replication lag. Sustained lag above 10 MB warrants investigation; the passive node may be unable to keep up.
  • Check /storage/archive only if on vCenter 6.5. In 6.7 and later, 100% on /storage/archive is normal by design.

How Netdata helps

  • Per-second disk utilization on /storage/db. A partition that grows from 70% to 95% during a single incident (event storm, log bomb) is visible in real time rather than after a 5-minute polling interval that misses the spike.
  • Correlate disk fill with service state. When /storage/db crosses 95%, the vpxd auto-shutdown and subsequent vPostgres crash produce correlated signals across disk utilization, service health, and API availability. Seeing them together confirms the causal chain without guessing.
  • Track database size and growth rate. A sustained growth trend on the vPostgres data directory, plotted over days and weeks, gives runway estimates before the cliff.
  • Alert before the 95% threshold. Alerts at 70% and 85% on /storage/db give operators time to reduce statistics level, trigger a purge, or plan a partition expansion before the auto-shutdown fires.
  • Surface dead tuple ratio and WAL accumulation. These internal vPostgres signals provide 30 to 60 minutes of warning before a database-related outage, before the partition reaches 95%.