Huge Pages on Netdata

Understanding Huge Pages

Thu, 04 May 2023 00:00:00 +0000

Memory-intensive applications can benefit from improved performance by using huge pages, as they can reduce TLB pressure and memory fragmentation, and lower the memory management overhead overall. Developers should consider using HugeTLBfs in their mmap() and shmget() calls to take advantage of huge pages.

Transparent Huge Pages (THP) is a Linux kernel feature that provides some of the benefits of huge pages without requiring any development effort. However, THP can cause latency in many applications. Although kernel developers are actively working to address these issues, many system administrators prefer to disable THP altogether.

MySQL OOM-killed: buffer pool, per-connection buffers, and the kernel killer

Mon, 01 Jan 0001 00:00:00 +0000

MySQL disappears. Application logs fill with connection timeouts. Uptime resets to near zero. In the kernel log: oom-kill: task mysqld and its anonymous RSS. Orchestrators mark the pod OOMKilled (exit code 137). To the application, this is a crash. To the kernel, mysqld was the largest memory consumer and the system ran out.

OOM kills are gradual, then catastrophic. Memory pressure builds as connections open, temp tables materialize, and dirty pages accumulate. The buffer pool is the obvious consumer, but the killer often enters through the back door: a connection burst multiplies per-thread buffers, a container limit sits too close to the buffer pool size, or Transparent Huge Pages block reclamation. Map all allocators to prevent recurrence.

Redis latest_fork_usec too high: THP, NUMA, and fork latency

Mon, 01 Jan 0001 00:00:00 +0000

INFO stats shows latest_fork_usec in the hundreds of milliseconds. Every fork() blocks the single event loop, so during that window no commands are processed. Clients time out, replicas disconnect, and a full resync can trigger another fork, creating a loop of latency and reconnection storms. A normal fork costs roughly 10-20ms per gigabyte of resident memory with Transparent Huge Pages disabled. If you are seeing 10-100x that, the culprit is usually THP, NUMA, or memory overcommit policy.

Redis OOM-killed by the kernel: RSS, overcommit, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Redis reports used_memory at 60% of maxmemory, then disappears. The container status is OOMKilled, or dmesg shows the kernel OOM killer selected redis-server. The kernel enforces resident memory (RSS), while used_memory and maxmemory track logical allocator state. Fragmentation, copy-on-write pages during persistence, and client buffers inflate RSS above the logical figure most operators monitor. When RSS hits the host or cgroup memory ceiling, the kernel terminates the process even though Redis believes it is within limits.

Redis sync_full incrementing: diagnosing full resync events

Mon, 01 Jan 0001 00:00:00 +0000

Your Redis primary’s sync_full counter is climbing. That means replicas are performing full resyncs instead of partial ones. Each full resync forces the primary to fork, write an RDB snapshot, and push it to the replica, which then wipes its own dataset and reloads from scratch. One full resync is a heavy operation. Several in succession, or multiple at once, can freeze the primary’s event loop, spike memory via copy-on-write, and trigger a cascade where more replicas fall behind and also need full resyncs.