vCenter /storage/log full: the log-bomb disk death spiral

You log in to the vSphere Client and get a 503, or the UI hangs mid-task. You SSH into the VCSA and run df -h: /storage/log is at 100%. The root filesystem may still have plenty of free space, which is why a generic disk-space alert missed it. The VCSA has many dedicated partitions, and they fill independently.

A single failing service can write gigabytes of logs per hour. STS authentication failures, database connection errors, alarm flapping, or a misbehaving SDK client flooding vpxd with errors will take /storage/log from 40% to 100% within hours. Once the partition is full, services that try to log crash. vmon restarts them. The restart itself generates more log lines as the service hits the same fault and tries to log it again. The loop is self-reinforcing, and clearing space temporarily makes the next iteration worse because the service can write again, refilling the partition faster.

This guide covers the failure pattern, how to identify the offending log source, and the correct way to truncate logs (not delete them) without losing the diagnostic evidence you need to fix the underlying cause.

What this means

The /storage/log partition on the VCSA holds logs for every service under vmware-vmon control: vpxd, STS, vPostgres, rhttpproxy, content-library, sps, perfcharts, and roughly two dozen more. Each service writes to its own subdirectory. When the partition hits 100%, every write fails. Some services handle that gracefully. Many crash.

The defining characteristic of a log bomb is that the disk-full state is caused by the same fault the logs are reporting. STS cert rejection causes authentication errors. Authentication errors are logged at high rate. The logs fill the disk. STS crashes because it cannot write logs. vmon restarts STS. STS logs its startup banner, hits the same auth fault, and writes more. Truncating the log file gives you hours of runway, but the partition refills unless you fix the underlying fault.

flowchart TD
    A[Underlying fault:
STS auth, DB connect, alarm flap] --> B[Service logs errors at GB/hour]
    B --> C[/storage/log fills to 100%]
    C --> D[Services cannot write logs]
    D --> E[Service crashes]
    E --> F[vmon restarts service]
    F --> G[Restart re-encounters same fault]
    G --> B
    C --> H[vpxd and other services crash]
    H --> I[503 errors, slow UI, management plane impaired]

A related and commonly missed failure mode is inode exhaustion. The partition reports 40% space used but writes still fail with ENOSPC because every rotated log fragment consumes an inode. Always run df -i alongside df -h.

Common causes

Quick checks

Run these read-only from the VCSA shell as root.

# Identify which partition is full
df -h

# Inode exhaustion can hit before space exhaustion
df -i

# Find the largest consumers under /storage/log
du -shx /storage/log/vmware/* 2>/dev/null | sort -h | tail -20

# Drill into the top offender
du -shx /storage/log/vmware/<service>/* 2>/dev/null | sort -h | tail -20

# Service status and restart state
/usr/lib/vmware-vmon/vmon-cli --list

# Errors in vpxd in the last hour
grep -c -i "error" /var/log/vmware/vpxd/vpxd.log

# SSO authentication failures
grep -c -E "LOGIN_FAILED|Authentication.*failed" /var/log/vmware/sso/vmware-sts-idmd.log

# Functional probe of vpxd through rhttpproxy
# 503 = vpxd unavailable; 401 or 200 = service is responding
curl -sk -o /dev/null -w "%{http_code}\n" https://localhost/sdk

Use -x on du to prevent it from crossing into other mounts via symlinks. Redirect stderr to suppress noise from special files and broken symlinks.

How to diagnose it

1. Confirm the partition. Run df -h and note which /storage/* mount is at 100%. Do not assume /storage/log. /storage/db, /storage/seat, /storage/core, and /storage/updatemgr can also fill, with overlapping symptoms.
2. Identify the offending service. Sort subdirectories of /storage/log/vmware/ by size. One service will typically dominate by an order of magnitude.
3. Identify the offending log file. Within that service directory, find the file growing fastest. ls -la repeated at 30-second intervals is often faster than du and does not walk the whole tree.
4. Read the tail of that file. The last 200 lines almost always reveal the underlying fault: STS cert rejection, ODBC connection refused, repeated managed-object reference, certificate validation error. Capture this evidence before truncating.
5. Check vmon state. /usr/lib/vmware-vmon/vmon-cli --list shows whether the service is STARTED, STOPPED, or FAILED, and whether vmon has hit its max-restart limit. A service in FAILED state after many restarts has been abandoned by vmon and will not recover on its own.
6. Check for inode exhaustion. Run df -i. If IUse% is near 100% on /storage/log, you have too many files, not too much data. Look for thousands of rotated fragments or state files under /storage/log/vmware/.
7. Check the upstream fault domain. Correlate with certificate expiry (/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store <store>), NTP offset (chronyc tracking), vPostgres health (service-control --status vmware-vpostgres), and AD/LDAP reachability.

Metrics and signals to monitor

Fixes

Truncate, never delete

The first action is to free space on /storage/log so services can write again. Use truncation, not deletion.

# Truncate the offending log file in place. The writing process keeps
# its file descriptor; space is released immediately.
> /storage/log/vmware/<service>/<logfile>
# or equivalently:
truncate -s 0 /storage/log/vmware/<service>/<logfile>

Never rm an open log file. The writing process holds the file descriptor, the kernel keeps the inode and the blocks allocated until that descriptor closes, and df continues to show 100% full even though ls no longer lists the file. This is one of the most common mistakes during a log-bomb incident and it makes diagnosis harder, not easier.

Before truncating, copy the last few hundred lines somewhere off-appliance if another partition has space. The tail is usually the only evidence of the underlying fault.

Stop the underlying log source

Truncation buys hours, not a fix. The partition refills as soon as the service hits the same error again. Once you have the evidence from the tail of the log, address the fault:

• STS cert expiry: follow the STS signing certificate renewal procedure. This is non-trivial and may require a maintenance window.
• AD/LDAP connectivity: restore DNS and network reachability to domain controllers.
• NTP skew: restore NTP synchronization (chronyc tracking) before restarting STS, or the auth errors will resume immediately.
• vpxd error flood from an external SDK client: identify the client by source IP and session count, then throttle, rate-limit, or temporarily block it.
• Alarm flapping: disable the alarm that is oscillating, then fix the underlying host or datastore condition.
• Crash loop abandoned by vmon: after fixing the underlying fault, manually start the service with service-control --start <service>.

Inode exhaustion

If df -i shows IUse% near 100%, truncation does not help. You need to remove files, not free bytes. Look for directories containing thousands of small rotated fragments:

# Find directories with the most files under /storage/log
find /storage/log -xdev -type d \
  -exec sh -c 'echo "$(ls -1 "$1" | wc -l) $1"' _ {} \; \
  | sort -rn | head -20

Rotate-and-compress policies that produce thousands of .gz fragments are the usual culprit. Removing the older fragments is safe because they are closed files, not held open by a writer.

Resize the partition

If /storage/log is chronically undersized for your environment, expand the VMDK and grow the LVM volume. The VCSA partitions are sized at deployment time based on deployment size (Tiny, Small, Medium, Large, X-Large), and an environment that has grown past its original sizing needs the partition grown to match.

A snapshot on the vCenter VM blocks VMDK growth. Remove snapshots before resizing.

Do not restart services first

Resist the urge to bounce vpxd or vPostgres to “clear the issue.” If /storage/log is still full, the restart generates more log lines (startup banners, inventory cache rebuild messages) and accelerates the spiral. Always truncate first, identify the cause, then restart only the specific service that was crash-looping.

Prevention

• Monitor every /storage/* partition independently, not just root. Alert on each.
• Alert on growth rate, not just absolute level. A partition going from 30% to 60% in an hour is an active incident even if 60% is below threshold.
• Include df -i in monitoring. Treat IUse% above 70% as a warning.
• Track vmon restart count per service. Any service restarting more than once per 15 minutes is in a crash loop.
• Track STS signing certificate expiry with a 60-day warning lead time. The renewal procedure is complex enough that shorter runway is an emergency, not a plan.
• Monitor NTP offset and SDK authentication failure rate. Both are leading indicators for the most common log-bomb triggers.
• Right-size the VCSA. A Tiny or Small appliance running a Large inventory fills /storage/log faster and recovers more slowly because vpxd has less headroom.

How Netdata helps

Netdata’s per-second metrics let you catch the partition-fill event and correlate it with the upstream cause, provided the right signals are collected from the VCSA guest.

• Per-partition disk utilization on every /storage/* mount surfaces the specific partition filling, not just root. A log bomb on /storage/log is visible even when / looks healthy.
• Disk space rate-of-change flags the leading indicator: a partition filling at 5 GB/hour is an active incident even if absolute utilization is still below threshold.
• Inode utilization is tracked alongside space utilization, so inode exhaustion from thousands of rotated fragments is caught before it looks like a disk-full incident.
• Service-level metrics for vpxd, vPostgres, STS, and rhttpproxy correlate the failing service with the filling partition.
• Anomaly advisories on log volume and error rate can flag the start of a log-bomb spiral before /storage/log crosses the page threshold.
• Correlation across signals ties the partition-fill event to the upstream cause: NTP drift, certificate expiry, or a new SDK client appearing just before the log rate spiked.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere DRS not balancing: affinity rules and reservations blocking placement
• vSphere DRS thrashing: vMotion churn with no stable placement
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong