vCenter /storage/seat full: stats, events, alarms, and tasks outgrowing their partition

The /storage/seat partition on the vCenter Server Appliance (VCSA) holds the vPostgres tables for Stats, Events, Alarms, and Tasks: vpx_event, vpx_event_arg, vpx_task, and the vpxd_hist_stat* rollup tables. In modern VCSA it is a dedicated mount, so it can fill while /storage/db, /storage/log, and / all show healthy utilization. Operators checking only / or the VAMI dashboard’s “VCDB” usage will miss it until vpxd refuses to start.

When /storage/seat crosses 95% utilization, vpxd refuses to come up to avoid database corruption. Without vCenter: DRS stops scheduling, vMotion is gone, HA cannot be reconfigured, no provisioning, no management operations. VMs on ESXi hosts keep running because the data plane is independent of vCenter, but everything that touches vCenter is broken.

The two dominant root causes: someone set statistics level 3 or 4 “for troubleshooting” weeks ago and never reverted it, or the periodic event/task purge job has fallen behind or stopped while vpxd was busy or down. Less common but documented: alarm or login storms generating millions of vpx_event rows per day, and on vSphere 8 with Tanzu, AVI load-balancer events from a full Supervisor Control Plane VM flooding the SEAT tables.

The operator trap that catches everyone the first time: purging old rows does not shrink the underlying filesystem. vPostgres reclaims space inside the table files for reuse, but does not return blocks to the filesystem without VACUUM FULL (which takes an ACCESS EXCLUSIVE lock and needs equivalent free space). The partition can stay at 95%+ for hours or days after you fix retention. Plan for that.

Failure cascade

As /storage/seat utilization rises, vPostgres queries on the SEAT tables slow from bloat, dead tuples, and checkpoint pressure. The vSphere Client gets sluggish. Statistics rollup jobs fall behind, leaving gaps in historical charts. At 95%, vpxd hits the hard stop and will not start.

flowchart TD
    A["Stats level 3/4 left on
OR purge job stalled"] --> B["SEAT tables grow:
vpx_event, vpx_task, vpxd_hist_stat*"]
    B --> C["/storage/seat utilization climbs"]
    C --> D{"Partition >= 95%?"}
    D -- No --> E["vpxd slows, UI sluggish,
DRS lagging"]
    D -- Yes --> F["vpxd refuses to start:
DB write guard"]
    F --> G["No DRS, no HA mgmt,
no provisioning"]
    G --> H["VMs keep running on hosts,
but unmanageable"]

Common causes

Quick checks

# SSH to the VCSA as root. All checks below are read-only.
# SEAT tables live in the 'vc' schema. If queries return no rows, prefix
# table names with vc. or run: SET search_path TO vc;

# 1. Confirm /storage/seat is the partition at fault
df -h /storage/seat

# 2. Compare to sibling partitions
df -h | grep -E 'seat|/storage/db|/storage/log'

# 3. Inode usage can hit before space on heavily-rowed tables
df -i /storage/seat

# 4. Total VCDB size
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c \
  "SELECT pg_size_pretty(pg_database_size('VCDB'));"

# 5. Top 20 largest tables (SEAT tables should dominate)
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "
SELECT schemaname, tablename,
       pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS total_size
FROM pg_tables
WHERE schemaname NOT IN ('pg_catalog','information_schema')
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 20;"

# 6. Approximate row counts for the SEAT tables (avoid COUNT(*) on large tables)
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "
SELECT relname, n_live_tup
FROM pg_stat_user_tables
WHERE relname IN ('vpx_event','vpx_event_arg','vpx_task')
ORDER BY n_live_tup DESC;"

# 7. Statistics and retention configuration
# TODO: verify parameter names; statistics level may be in vpx_stat_level, not vpx_parameter
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c \
  "SELECT * FROM vc.vpx_parameter WHERE name LIKE '%stats%' OR name LIKE '%maxAge%';"

# 8. Newest sample in each stats rollup level (catches rollup lag)
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "
SELECT 'level1' AS lvl, MAX(sample_time) FROM vc.vpxd_hist_stat1
UNION ALL SELECT 'level2', MAX(sample_time) FROM vc.vpxd_hist_stat2
UNION ALL SELECT 'level3', MAX(sample_time) FROM vc.vpxd_hist_stat3
UNION ALL SELECT 'level4', MAX(sample_time) FROM vc.vpxd_hist_stat4;"

# 9. Top event types in the last 24 hours (storm detection)
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "
SELECT event_type, count(*) AS rows
FROM vc.vpx_event
WHERE create_time > now() - interval '24 hours'
GROUP BY event_type
ORDER BY rows DESC LIMIT 10;"

# 10. Dead tuple ratio on major tables (autovacuum effectiveness)
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "
SELECT relname, n_live_tup, n_dead_tup,
       round(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) AS dead_pct,
       last_autovacuum
FROM pg_stat_user_tables
WHERE relname IN ('vpx_event','vpx_event_arg','vpx_task')
ORDER BY n_dead_tup DESC;"

# 11. vpxd log entries indicating the SEAT threshold was hit
grep -i "storage.*seat\|exceeds.*threshold\|space.*partition" \
  /var/log/vmware/vpxd/vpxd.log | tail -30

How to diagnose it

1. Confirm the partition. Quick check #1 shows whether /storage/seat is the culprit. If a different /storage/* mount is full, this is a different failure pattern (log bomb on /storage/log, WAL accumulation on /storage/db in VCHA setups, core dump accumulation on /storage/core).
2. Confirm vpxd is the affected service. Run service-control --status and look for vmware-vpxd. If STOPPED with a storage threshold message in vpxd.log, SEAT exhaustion is confirmed.
3. Identify the dominant tables. Quick check #5 shows whether events, tasks, or stats rollup tables dominate. Events point to a retention or storm problem; stats rollup tables point to statistics level.
4. Verify statistics level and retention. Quick check #7 exposes configured levels and retention windows. If level is 3 or 4, that is almost certainly the root cause. If event/task retention is set to months or years, that is the cause.
5. Check for an event storm. Quick check #9 exposes storm patterns: tens of thousands of StatelessAlarmTriggeredEvent, AuthFailed, or repeated host events. Address the underlying source before purging or the partition will refill within days.
6. Check purge job progress. The purge runs as an internal vpxd task. If vpxd was down for an extended period, no purge happened. After restarting vpxd, watch n_live_tup on vpx_event/vpx_task over an hour; counts should drop.

Metrics and signals to monitor

Fixes

Reduce statistics level

In the vSphere Client: Administration > vCenter Server Settings > Statistics. Set every interval back to Level 1 unless a product genuinely requires higher (Aria Operations, third-party dashboards). The change takes effect for new samples; existing rollup data is still subject to retention.

This is the highest-leverage, lowest-risk fix. Apply it before purging if SEAT is dominated by vpxd_hist_stat* tables.

Reduce retention (events and tasks)

In the vSphere Client: Administration > vCenter Server Settings > Database Retention Policy. Default event/task retention is 30 days. If retention was raised to 180 or 365 days “for audit,” reduce it back to 30 and rely on an external syslog or SIEM for long-term retention. vCenter’s database is not an event archive.

Run the KB2110031 purge scripts

When retention reduction is not enough, or vpxd cannot start, use the VMware-provided purge scripts documented in KB2110031. These scripts delete old events, tasks, and statistics rows past a configurable cutoff. This is permanent data deletion. Take a snapshot or backup of the VCSA before running.

If /storage/seat is at 100%, vPostgres may not be able to write WAL even to execute the purge. Grow the partition first (see below). Engage VMware support if you are uncertain which script applies to your version.

Grow the partition with autogrow.sh

To expand /storage/seat:

1. Power off the VCSA VM.
2. Edit the VMDK that maps to the SEAT partition. The disk number depends on deployment size and vSphere version; reference the VCSA virtual disk layout for your version before editing.
3. Power the VCSA back on.
4. As root, run /usr/lib/applmgmt/support/scripts/autogrow.sh. The script detects the new size and extends the LVM volume.

autogrow.sh only grows. It does not shrink partitions after a purge, and root partition resizing is not supported on vCenter 7.0 and above.

Address the underlying event storm

If you found a flood of alarm, auth, or AVI events, fix the source before purging or the partition will refill within days. Common patterns:

• Alarm threshold flapping. An alarm fires and clears on hundreds of VMs in a loop, generating millions of StatelessAlarmTriggeredEvent rows. Tune the trigger or disable the alarm.
• SSH brute force against an ESXi host. Generates large volumes of AuthFailed events. Restrict source IPs, disable SSH when not in use, or enable lockdown mode.
• vSphere 8 with Tanzu. A full Supervisor Control Plane VM produces excessive AVI load-balancer login events that flood vpx_event. Resolve the Supervisor VM disk exhaustion first.
• Misbehaving integration. A monitoring, backup, or orchestration client polling too aggressively can generate event and task volume that dwarfs normal operations. Throttle the client or add backoff.

Reclaim filesystem space (last resort)

After purge and retention fixes, the partition may still show high utilization because vPostgres holds the freed space inside the table files for reuse. Options:

• Wait. New writes reuse the freed pages internally. Effective, but slow if write volume is low.
• VACUUM FULL on the largest SEAT tables. Rewrites the table and returns space to the filesystem. Takes an ACCESS EXCLUSIVE lock, blocks all operations on that table, and requires free space roughly equal to the table to complete. Plan a maintenance window and validate with VMware support before running on production vCenter.
• Recreate the tablespace. More invasive and unsupported without VMware guidance. Engage support.

Prevention

• Statistics level discipline. Default to Level 1. Document any deviation with a ticket reference and a revert-by date. Audit quarterly.
• Retention policy aligned to operational use. Keep the 30-day default for events and tasks unless compliance demands otherwise. Move long-term audit retention to an external syslog or SIEM.
• Per-partition disk monitoring on the VCSA. /storage/seat is its own mount. Monitoring only / or /storage/db will miss this incident. Include every /storage/* mount: core, updatemgr, lifecycle, db, log.
• Growth-rate alerting, not just threshold. Alert when daily growth exceeds 500MB on a stable inventory. Threshold-only alerting at 80% can fire too late to prevent the 95% hard stop.
• Alarm tuning review. Periodically check the top event types by count. A single alarm dominating is a leading indicator of an impending SEAT incident.
• vpxd uptime hygiene. Long vpxd outages mean the purge job did not run. Track vpxd restarts and total downtime.

How Netdata helps

• Per-mount filesystem utilization on the VCSA, including /storage/seat as its own series, so a SEAT-only fill is visible even when sibling partitions look healthy.
• Per-second utilization and growth-rate charts, letting you correlate a statistics level change or an event storm with the moment the SEAT partition started climbing.
• Postgres internal metrics (database size, table sizes, dead tuple ratio, connection count) alongside filesystem metrics, so you can see whether the database is bloated independent of the partition.
• Service-state tracking for vmware-vpxd and vmware-vpostgres, with anomaly detection that catches the brief silent restarts that precede the full crash.
• Anomaly detection on log volume from /var/log/vmware/vpxd/, which often spikes hours before the partition fills.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere DRS not balancing: affinity rules and reservations blocking placement
• vSphere DRS thrashing: vMotion churn with no stable placement
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong