Redis latest_fork_usec too high: THP, NUMA, and fork latency

Mon, 01 Jan 0001 00:00:00 +0000

INFO stats shows latest_fork_usec in the hundreds of milliseconds. Every fork() blocks the single event loop, so during that window no commands are processed. Clients time out, replicas disconnect, and a full resync can trigger another fork, creating a loop of latency and reconnection storms. A normal fork costs roughly 10-20ms per gigabyte of resident memory with Transparent Huge Pages disabled. If you are seeing 10-100x that, the culprit is usually THP, NUMA, or memory overcommit policy.

vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty

Mon, 01 Jan 0001 00:00:00 +0000

%CSTP in esxtop is the time a vCPU in a multi-vCPU VM sits halted because the ESXi scheduler is waiting to co-schedule the VM’s other vCPUs. In a healthy environment it is essentially zero. Sustained above a few percent on modern ESXi means a sizing or topology problem, not a performance problem you can tune away.

The classic shape: you give a database 16 vCPUs and it gets slower. The guest OS reports low CPU utilization because the vCPUs are not doing work. They are parked in COSTOP waiting for their siblings. From inside the VM this is invisible. The application runs slowly while the OS reports idle capacity.

NUMA Architecture on Netdata

Redis latest_fork_usec too high: THP, NUMA, and fork latency

vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty