<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Memory on Netdata</title><link>https://www.netdata.cloud/tags/memory/</link><description>Recent content in Memory on Netdata</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Sun, 23 Aug 2026 02:03:02 +0300</lastBuildDate><atom:link href="https://www.netdata.cloud/tags/memory/index.xml" rel="self" type="application/rss+xml"/><item><title>Linux Cgroups V2 Memory Throttling &amp; OOM Fix</title><link>https://www.netdata.cloud/academy/diagnosing-linux-cgroups/</link><pubDate>Fri, 05 Sep 2025 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/academy/diagnosing-linux-cgroups/</guid><description>&lt;p&gt;Your critical service is lagging. Users are complaining about timeouts. You check your orchestration platform and see the dreaded &lt;code&gt;OOMKilled&lt;/code&gt; status on a container. You dive into the node&amp;rsquo;s logs (&lt;code&gt;dmesg&lt;/code&gt;) and confirm it: the kernel&amp;rsquo;s Out-of-Memory (OOM) killer has claimed another victim. The immediate fix is easy—restart the container, maybe give it more memory—but the real question remains unanswered: &lt;em&gt;why&lt;/em&gt; did it happen? Was it a sudden memory leak, a traffic spike, or something more subtle?&lt;/p&gt;</description></item><item><title>How To Find And Fix Memory Leaks in C or C++</title><link>https://www.netdata.cloud/academy/how-to-find-memory-leak-in-c/</link><pubDate>Tue, 10 Jun 2025 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/academy/how-to-find-memory-leak-in-c/</guid><description>&lt;p&gt;Your application feels sluggish. It runs perfectly after a restart, but over hours or days, it slows to a crawl before eventually crashing. If you&amp;rsquo;re working with C or C++, this behavior is a classic symptom of a memory leak—a silent bug that can drain system resources and destabilize your services. Because these languages put memory management directly in your hands, understanding how to find and fix memory leaks is a critical skill for building robust, long-running applications.&lt;/p&gt;</description></item><item><title>What Is A Memory Leak In Java How To Detect And Fix Them</title><link>https://www.netdata.cloud/academy/java-memory-leak/</link><pubDate>Tue, 10 Jun 2025 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/academy/java-memory-leak/</guid><description>&lt;p&gt;Your Java application runs smoothly after a fresh deploy, but over hours or days, its performance steadily degrades. Response times creep up, garbage collection pauses become longer and more frequent, and then, the inevitable happens: the application crashes, logging a fatal OutOfMemoryError. This classic scenario is often the calling card of a subtle but dangerous problem—a memory leak.&lt;/p&gt;&#10;&lt;p&gt;Even though Java features automatic memory management via its garbage collector (GC), applications are not immune to leaks. A Java memory leak occurs when objects are no longer in use by the application, but the GC is unable to reclaim their memory because they are still being referenced. Over time, these orphaned objects accumulate, consuming the available heap space and leading to performance degradation and eventual failure. Understanding how to detect and fix these leaks is a critical skill for any Java developer.&lt;/p&gt;</description></item><item><title>Garbage Collection In Java What It Is and How It Works</title><link>https://www.netdata.cloud/academy/java-garbage-collection/</link><pubDate>Mon, 09 Jun 2025 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/academy/java-garbage-collection/</guid><description>&lt;p&gt;One of the most powerful features of the Java platform is its automatic memory management. Unlike languages like &lt;a href="https://www.netdata.cloud/academy/how-to-find-memory-leak-in-c/"&gt;C or C++&lt;/a&gt;, where developers must manually allocate and deallocate memory, Java handles this process for you through a process called garbage collection (GC). This frees developers to focus on application logic rather than the complexities of memory management, which is a major reason for Java&amp;rsquo;s enduring popularity.&lt;/p&gt;&#10;&lt;p&gt;But what exactly is garbage collection in Java, and how does it work under the hood? While it&amp;rsquo;s an automatic process, a solid understanding of the Java garbage collector is crucial for writing high-performance, stable applications and for troubleshooting &lt;a href="https://www.netdata.cloud/academy/java-memory-leak/"&gt;memory-related issues&lt;/a&gt; like the dreaded &lt;code&gt;OutOfMemoryError&lt;/code&gt;.&lt;/p&gt;</description></item><item><title>Nodejs Memory Leak How To Identify Debug And Avoid Them</title><link>https://www.netdata.cloud/academy/nodejs-memory-leak/</link><pubDate>Mon, 26 May 2025 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/academy/nodejs-memory-leak/</guid><description>&lt;p&gt;In the fast-paced world of Node.js development, performance and reliability are non-negotiable. However, a silent saboteur often lurks in the shadows – the Node.js memory leak. These insidious issues can gradually degrade your application&amp;rsquo;s performance, leading to slowdowns, crashes, and frustrated users. Understanding how to effectively identify, debug, and &lt;a href="https://www.netdata.cloud/academy/how-to-find-memory-leak-in-c/"&gt;prevent memory leaks&lt;/a&gt; is a critical skill for any developer, DevOps engineer, or SRE working with Node.js. This guide will walk you through the intricacies of Node.js memory management and equip you with the knowledge to tackle these challenging problems.&lt;/p&gt;</description></item><item><title>Key Observability Metrics | Infrastructure &amp; APM Monitoring</title><link>https://www.netdata.cloud/academy/a-guide-to-the-most-important-observability-metrics/</link><pubDate>Thu, 06 Jun 2024 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/academy/a-guide-to-the-most-important-observability-metrics/</guid><description>&lt;h2 id="what-is-observability-the-fundamentals"&gt;What Is Observability? The Fundamentals&lt;/h2&gt;&#10;&lt;p&gt;Noone can argue that observability is crucial for maintaining the health and performance of applications and infrastructure. Observability refers to the ability to measure and understand the state of a system based on the outputs it produces. This is extremely important for identifying, diagnosing, and resolving issues effectively and efficiently.&lt;/p&gt;&#10;&lt;p&gt;Observability is essential for DevOps and &lt;a href="https://www.ibm.com/think/topics/site-reliability-engineering" target="_blank"&gt;SRE&lt;/a&gt; teams as it provides a comprehensive, overall view of the infrastructure’s health, enabling proactive maintenance and quicker incident response. It involves collecting and analyzing a variety of data types, including logs, metrics, and traces, to gain insights into system behavior and it can help discover possible anomalies throughout the whole infrastructure.&lt;/p&gt;</description></item><item><title>Real-Time Insights for Reliable Web Hosting</title><link>https://www.netdata.cloud/case-studies/hosting/trillium/</link><pubDate>Thu, 01 Feb 2024 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/case-studies/hosting/trillium/</guid><description>&lt;h2 id="simplifying-monitoring-for-enhanced-performance"&gt;Simplifying Monitoring for Enhanced Performance&lt;/h2&gt;&#10;&lt;p&gt;Trillium was confronted with the challenge of finding an effective monitoring solution that could be easily integrated with their systems, despite the non-technical background of some team members. The company aimed to uphold its commitment to outstanding service uptime and performance—a critical aspect in the hosting industry. The search for a monitoring tool that could provide comprehensive insights into the health of their infrastructure, particularly RAM and CPU usage, and uptime tracking, led them to Netdata.&lt;/p&gt;</description></item><item><title>Understanding Huge Pages</title><link>https://www.netdata.cloud/blog/understanding-huge-pages/</link><pubDate>Thu, 04 May 2023 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/blog/understanding-huge-pages/</guid><description>&lt;p&gt;Memory-intensive applications can benefit from &lt;a href="https://www.netdata.cloud/academy/what-is-application-performance-monitoring-apm/"&gt;improved performance&lt;/a&gt; by using huge pages, as they can reduce TLB pressure and memory fragmentation, and lower the memory management overhead overall. Developers should consider using HugeTLBfs in their mmap() and shmget() calls to take advantage of huge pages.&lt;/p&gt;&#10;&lt;p&gt;Transparent Huge Pages (THP) is a Linux kernel feature that provides some of the benefits of huge pages without requiring any development effort. However, THP can cause latency in many applications. Although kernel developers are actively working to address these issues, many system administrators prefer to disable THP altogether.&lt;/p&gt;</description></item><item><title>Unlock the Secrets of Kernel Memory Usage</title><link>https://www.netdata.cloud/blog/unlock-the-secrets-of-kernel-memory-usage/</link><pubDate>Thu, 04 May 2023 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/blog/unlock-the-secrets-of-kernel-memory-usage/</guid><description>&lt;p&gt;&lt;img src="../2023-05-04-unlock-the-secrets-of-kernel-memory-usage/img/stacked-netdata.png" alt="stacked-netdata"&gt;&lt;/p&gt;&#10;&lt;p&gt;The &lt;code&gt;mem.kernel&lt;/code&gt; chart in Netdata provides insight into the memory usage of &lt;a href="https://www.netdata.cloud/academy/what-are-the-differences-between-bpf-and-ebpf-an-overview/"&gt;various kernel subsystems&lt;/a&gt; and mechanisms. By understanding these dimensions and their technical details, you can monitor your system&amp;rsquo;s kernel memory usage and identify potential issues or inefficiencies. Monitoring these dimensions can help you ensure that your system is running efficiently and provide valuable insights into the performance of your kernel and memory subsystem.&lt;/p&gt;&#10;&lt;p&gt;&lt;img src="../2023-05-04-unlock-the-secrets-of-kernel-memory-usage/img/mem-kernel.png" alt="mem-kernel"&gt;&lt;/p&gt;&#10;&lt;!-- truncate --&gt;&#10;&lt;h2 id="slab"&gt;Slab&lt;/h2&gt;&#10;&lt;p&gt;The &lt;a href="https://en.wikipedia.org/wiki/Slab_allocation"&gt;slab allocator&lt;/a&gt; is a memory management mechanism introduced by Jeff Bonwick in 1994 to manage &lt;a href="https://www.netdata.cloud/monitoring-101/vsphere-monitoring/"&gt;memory allocation&lt;/a&gt; for kernel objects. The main purpose of the slab allocator is to reduce memory fragmentation and improve the speed of memory allocation/deallocation. The slab allocator groups objects of the same size into &amp;ldquo;slabs&amp;rdquo; and caches the objects to speed up future allocations.&lt;/p&gt;</description></item><item><title>Swap Memory: When &amp; How To Use It On Production VMs</title><link>https://www.netdata.cloud/blog/swap-memory-when-and-how-to-use-it-on-your-production-systems-or-cloud-provided-vms/</link><pubDate>Tue, 02 May 2023 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/blog/swap-memory-when-and-how-to-use-it-on-your-production-systems-or-cloud-provided-vms/</guid><description>&lt;p&gt;&lt;img src="../2023-05-02-swap-memory-when-to-use-in-production-systems/img/stacked-netdata.png" alt="Swap Memory: Its Use On Production Systems &amp;amp; Cloud-Provided VMs"&gt;&lt;/p&gt;&#10;&lt;p&gt;Swap memory, also known as virtual memory, is a space on a hard disk that is used to supplement the physical memory (RAM) of a computer. The swap space is used when the system runs out of physical memory, and it moves less frequently accessed data from RAM to the hard disk, freeing up space in RAM for more frequently accessed data. But should swap memory be enabled on production systems and &lt;a href="https://www.netdata.cloud/monitoring-101/vsphere-monitoring/"&gt;cloud-provided virtual machines&lt;/a&gt; (VMs)? Let&amp;rsquo;s explore the pros and cons.&lt;/p&gt;</description></item><item><title>Monitor KSM performance with Netdata</title><link>https://www.netdata.cloud/blog/ksm/</link><pubDate>Tue, 01 Nov 2022 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/blog/ksm/</guid><description>&lt;p&gt;Monitoring KSM (Kernel Same-page Merging) performance at deduping memory shared across VMs.&lt;/p&gt;&#10;&lt;!--truncate--&gt;&#10;&lt;h2 id="kernel-same-page-merging-ksm"&gt;Kernel Same-page Merging (KSM)&lt;/h2&gt;&#10;&lt;p&gt;Linux kernels store memory in &lt;strong&gt;pages&lt;/strong&gt; which are moved in and out of memory as a single block. On most Linux architectures pages are 4096 bytes. &lt;strong&gt;KSM&lt;/strong&gt; (Kernel Same-page Merging) is a kernel feature that scans memory looking for pages with identical content, and then de-duplicates them. The most common use-case where such duplicate pages occur is on hosts running multiple virtual machines (VMs).&lt;/p&gt;</description></item><item><title>Netdata v1.31.0</title><link>https://www.netdata.cloud/blog/netdata-v1-31/</link><pubDate>Wed, 19 May 2021 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/blog/netdata-v1-31/</guid><description>&lt;!--truncate--&gt;&#10;&lt;img class="alignnone size-medium wp-image-16402" src="../wp-archive/uploads/2021/05/v1.31.01-1-600x375.png" alt="" width="600" height="375" /&gt;&#10;&lt;p&gt;Give a warm welcome to Netdata v1.31.0, which features:&lt;/p&gt;&#10;&lt;ul&gt;&#10; &#9;&lt;li aria-level="1"&gt;&lt;strong&gt;Re-packaged and redesigned dashboard&lt;/strong&gt;: A more informational and feature-rich “frame” for your monitoring and troubleshooting sessions.&lt;/li&gt;&#10; &#9;&lt;li aria-level="1"&gt;&lt;strong&gt;eBPF expands into the directory cache&lt;/strong&gt;: Monitor whether your services or applications are properly using Linux’s memory management for the best performance and minimal disk I/O.&lt;/li&gt;&#10; &#9;&lt;li aria-level="1"&gt;&lt;strong&gt;Machine learning-powered collectors&lt;/strong&gt;: Detect anomalies using only your own data and minimal resource utilization on your monitored nodes.&lt;/li&gt;&#10; &#9;&lt;li aria-level="1"&gt;&lt;strong&gt;An improved Netdata learning experience&lt;/strong&gt;: A timeline of new content, refreshed visuals, and a newly-open sourced repository.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h2 id="h_1876248811621353710666"&gt;Re-packaged and redesigned dashboard&lt;/h2&gt;&#10;We re-packaged and redesigned portions of the dashboard to improve the overall experience. Part of this effort is better handling of dashboard code during installation—anyone using third-party packages (such as the Netdata Homebrew formula) will start seeing new features and the new designs starting today.&#10;&lt;p&gt;For those who aren’t using third-party packages (thank you!), your installation process will still get a little bit faster.&lt;/p&gt;</description></item><item><title>Cassandra GC death spiral: long pauses, gossip flapping, and recovery</title><link>https://www.netdata.cloud/guides/cassandra/cassandra-gc-death-spiral/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/cassandra/cassandra-gc-death-spiral/</guid><description>&lt;p&gt;You are paged because a Cassandra node is flapping between UP and DOWN in &lt;code&gt;nodetool status&lt;/code&gt;, client timeouts are rising, and system logs show &lt;code&gt;GCInspector&lt;/code&gt; warnings. The node has not crashed. It is stuck in a GC death spiral: heap pressure produces long pauses, gossip marks the node DOWN, and the resulting retry and hint traffic creates even more heap pressure when the node recovers. It can start with a single large partition read, a misconfigured cache, or an oversized batch statement, and escalates until the node is effectively useless. Catch it early by watching the GC floor and gossip stability together, not just process uptime.&lt;/p&gt;</description></item><item><title>Cassandra GC pauses too long: diagnosing G1 stop-the-world pauses</title><link>https://www.netdata.cloud/guides/cassandra/cassandra-gc-pauses-too-long/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/cassandra/cassandra-gc-pauses-too-long/</guid><description>&lt;p&gt;&lt;code&gt;ReadTimeoutException&lt;/code&gt; and &lt;code&gt;WriteTimeoutException&lt;/code&gt; from clients, &lt;code&gt;GCInspector&lt;/code&gt; warnings in &lt;code&gt;system.log&lt;/code&gt;, and nodes flapping between &lt;code&gt;UP&lt;/code&gt; and &lt;code&gt;DOWN&lt;/code&gt; in &lt;code&gt;nodetool status&lt;/code&gt; without a JVM restart mean G1 is producing long stop-the-world pauses. Root causes include promotion pressure, humongous objects, or allocation bursts. Left unchecked, one node&amp;rsquo;s pauses trigger gossip failures, retries, and hint replay that drive cluster-wide degradation.&lt;/p&gt;&#10;&lt;h2 id="what-this-means"&gt;What this means&lt;/h2&gt;&#10;&lt;p&gt;G1GC is the default collector for Cassandra 4.x on JDK 11+. During a stop-the-world pause, every thread freezes, including gossip, native transport, and compaction. Cassandra logs &lt;code&gt;GCInspector&lt;/code&gt; warnings when a pause exceeds the configured threshold, commonly 500 ms. Pauses longer than ~2 seconds cause gossip rounds to be missed; under the default phi accrual failure detector threshold of 8, sustained pauses result in the node being marked DOWN by the failure detector. While the JVM is paused, mutations queue, reads stall, hints accumulate on peers, and clients retry. On recovery, hint replay and retry bursts raise allocation pressure, creating a self-reinforcing spiral.&lt;/p&gt;</description></item><item><title>Cassandra heap pressure: sizing the JVM heap and tuning G1GC</title><link>https://www.netdata.cloud/guides/cassandra/cassandra-heap-pressure-tuning/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/cassandra/cassandra-heap-pressure-tuning/</guid><description>&lt;p&gt;Cassandra runs as a single JVM process per node. Every write path allocation, memtable mutation, read merge buffer, and cache entry lives on the heap. When the heap is undersized or GC is left at JVM defaults, stop-the-world pauses freeze gossip, client requests, and compaction. A pause longer than roughly 18 seconds (the default phi accrual threshold is 8) causes peers to mark the node DOWN, which triggers hinted handoff, replay storms, and client retries that worsen memory pressure: the GC death spiral.&lt;/p&gt;</description></item><item><title>Cassandra java.lang.OutOfMemoryError: Java heap space - causes and recovery</title><link>https://www.netdata.cloud/guides/cassandra/cassandra-out-of-memory-error/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/cassandra/cassandra-out-of-memory-error/</guid><description>&lt;p&gt;The error &lt;code&gt;java.lang.OutOfMemoryError: Java heap space&lt;/code&gt; in Cassandra system logs means the JVM exhausted its allocated heap and could not satisfy an allocation request. The process typically exits, or the Linux OOM killer terminates it. Gossip marks the node DOWN, clients experience timeouts, and the remaining replicas absorb the orphaned traffic.&lt;/p&gt;&#10;&lt;p&gt;Unlike gradual GC pressure that degrades latency over hours, a heap-space OOM is often a hard stop. The node was serving traffic, then it was not. The challenge is not recognizing the failure (the error string is unambiguous) but finding which of several competing heap consumers caused it.&lt;/p&gt;</description></item><item><title>Cassandra killed by the Linux OOM killer: off-heap memory and RSS</title><link>https://www.netdata.cloud/guides/cassandra/cassandra-off-heap-oom-kill/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/cassandra/cassandra-off-heap-oom-kill/</guid><description>&lt;p&gt;The JVM heap chart shows 50% utilization and a flat line. There is no &lt;code&gt;OutOfMemoryError&lt;/code&gt;. Then the Cassandra process vanishes. &lt;code&gt;dmesg&lt;/code&gt; shows the OOM killer terminated the JVM: &lt;code&gt;Killed process 12345 (java)&lt;/code&gt;. The JVM heap metric does not include native allocations: bloom filters, compression metadata, index summaries, direct buffers, and chunk cache. When heap plus off-heap RSS exceeds available RAM, the kernel kills the process. This guide covers how to confirm that pattern, reduce off-heap footprint, and prevent recurrence.&lt;/p&gt;</description></item><item><title>ClickHouse killed by the OOM killer: RSS, max_server_memory_usage, and cgroup limits</title><link>https://www.netdata.cloud/guides/clickhouse/clickhouse-oom-killed-by-kernel/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/clickhouse/clickhouse-oom-killed-by-kernel/</guid><description>&lt;p&gt;You restart a pod and &lt;code&gt;kubectl describe pod&lt;/code&gt; shows &lt;code&gt;Reason: OOMKilled&lt;/code&gt; with exit code 137. Inside ClickHouse, &lt;code&gt;MemoryTracking&lt;/code&gt; sits well below &lt;code&gt;max_server_memory_usage&lt;/code&gt;, and &lt;code&gt;system.text_log&lt;/code&gt; shows no warning. The process is gone, merges are dead, and replication queues are backing up.&lt;/p&gt;&#10;&lt;p&gt;The Linux OOM killer targets RSS, not ClickHouse&amp;rsquo;s internal &lt;code&gt;MemoryTracking&lt;/code&gt;. Untracked allocations, jemalloc arena fragmentation, and cgroup accounting quirks create a persistent gap between what ClickHouse thinks it is using and what the kernel sees. In containerized environments, the cgroup OOM killer can evict the pod before ClickHouse ever triggers its own server-wide limit.&lt;/p&gt;</description></item><item><title>ClickHouse long-running queries: finding and killing the resource hog</title><link>https://www.netdata.cloud/guides/clickhouse/clickhouse-long-running-queries/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/clickhouse/clickhouse-long-running-queries/</guid><description>&lt;p&gt;A query that should finish in seconds is still running after twenty minutes. Memory on the ClickHouse node is climbing, query latency has doubled, and you suspect a single query is holding resources it will never release. In ClickHouse, a long-running query can be a legitimate analytical job crunching terabytes, a Cartesian JOIN exploding in memory, or a GROUP BY that has spilled to disk and slowed to a crawl. Telling the difference determines whether you kill it or let it finish.&lt;/p&gt;</description></item><item><title>ClickHouse Memory limit (for query) exceeded: per-query limits and GROUP BY/JOIN blowups</title><link>https://www.netdata.cloud/guides/clickhouse/clickhouse-memory-limit-for-query-exceeded/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/clickhouse/clickhouse-memory-limit-for-query-exceeded/</guid><description>&lt;p&gt;&lt;code&gt;Code: 241. DB::Exception: Memory limit (for query) exceeded&lt;/code&gt; means a single query&amp;rsquo;s allocations breached the &lt;code&gt;max_memory_usage&lt;/code&gt; ceiling. ClickHouse tracks memory in a hierarchy: server-wide, per-user, and per-query. The server kills the query to protect the rest of the workload.&lt;/p&gt;&#10;&lt;p&gt;This differs from &lt;code&gt;Memory limit (total) exceeded&lt;/code&gt; (server-level pressure) and &lt;code&gt;Memory limit (for user) exceeded&lt;/code&gt; (profile-level pressure). A query-level breach usually stems from one of three patterns: a high-cardinality &lt;code&gt;GROUP BY&lt;/code&gt;, an unbounded &lt;code&gt;DISTINCT&lt;/code&gt;, or a &lt;code&gt;JOIN&lt;/code&gt; that materializes more rows than expected.&lt;/p&gt;</description></item><item><title>ClickHouse Memory limit (total) exceeded - server-wide memory pressure and fixes</title><link>https://www.netdata.cloud/guides/clickhouse/clickhouse-memory-limit-total-exceeded/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/clickhouse/clickhouse-memory-limit-total-exceeded/</guid><description>&lt;p&gt;&lt;code&gt;Code: 241. DB::Exception: Memory limit (total) exceeded: would use X bytes, current RSS Y, maximum Z.&lt;/code&gt; is the server-level cap, not a per-query limit. When ClickHouse&amp;rsquo;s &lt;code&gt;MemoryTracking&lt;/code&gt; hits &lt;code&gt;max_server_memory_usage&lt;/code&gt; (default 90% of physical RAM), the server kills the heaviest running queries to protect the process. New and existing queries fail until memory drops. Find the largest memory consumer and stop it before the OOM killer does.&lt;/p&gt;&#10;&lt;p&gt;Unlike a per-query &lt;code&gt;max_memory_usage&lt;/code&gt; breach, the &lt;code&gt;(total)&lt;/code&gt; variant means the aggregate across all queries, background merges, caches, and dictionaries has crossed the server ceiling. It often arrives suddenly: a runaway JOIN, a burst of concurrent heavy &lt;code&gt;GROUP BY&lt;/code&gt; queries, or a background merge spike pushes the tracked total over the line. ClickHouse uses a hierarchical memory tracker (server to user to query), and the server limit is the final backstop.&lt;/p&gt;</description></item><item><title>ClickHouse memory pressure death spiral: runaway queries, retries, and OOM</title><link>https://www.netdata.cloud/guides/clickhouse/clickhouse-memory-pressure-death-spiral/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/clickhouse/clickhouse-memory-pressure-death-spiral/</guid><description>&lt;p&gt;&lt;code&gt;MEMORY_LIMIT_EXCEEDED&lt;/code&gt; errors climb in the query log. Queries that normally finish in seconds now take minutes or are killed outright. The ClickHouse process is near its memory limit, but killing the heaviest query only frees capacity for a moment before another query is killed. If the application retries immediately, pressure never drops. With spill-to-disk enabled, the bottleneck shifts to disk I/O, starving background merges and slowing the whole system.&lt;/p&gt;&#10;&lt;p&gt;This is the memory pressure death spiral: a composite failure pattern where memory saturation triggers query termination or spill-to-disk, rising latency provokes retries, and repeated attempts keep memory pinned near the limit. The spiral ends with an OS-level OOM kill if internal tracker and cgroup limits are misaligned, or with a merge crisis if temporary disk files consume all I/O bandwidth.&lt;/p&gt;</description></item><item><title>CockroachDB error 53200: SQL memory budget exhausted and query rejection</title><link>https://www.netdata.cloud/guides/cockroachdb/cockroachdb-sql-memory-budget-53200/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/cockroachdb/cockroachdb-sql-memory-budget-53200/</guid><description>&lt;p&gt;Error 53200 is CockroachDB&amp;rsquo;s PostgreSQL-compatible signal that the per-node SQL memory budget has run out. Applications see SQLSTATE 53200 (&amp;ldquo;insufficient resources&amp;rdquo;) with text such as &amp;ldquo;memory budget exceeded&amp;rdquo; and byte counts showing what was requested, what is allocated, and what the budget allows.&lt;/p&gt;&#10;&lt;p&gt;The SQL memory budget is enforced per-node, bounded by &lt;code&gt;--max-sql-memory&lt;/code&gt; (default 25% of system RAM). When a node&amp;rsquo;s SQL execution layer exhausts its budget, queries on that node either spill to temporary disk storage (slow) or are rejected outright with 53200. One large analytical query on a single gateway node can starve every other session connected to that node, even if the rest of the cluster has memory to spare.&lt;/p&gt;</description></item><item><title>CockroachDB memory pressure, GC thrashing, and Raft liveness failure</title><link>https://www.netdata.cloud/guides/cockroachdb/cockroachdb-memory-gc-liveness-thrash/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/cockroachdb/cockroachdb-memory-gc-liveness-thrash/</guid><description>&lt;p&gt;A CockroachDB node loses liveness in a repeating pattern: it drops out, the cluster redistributes its leases, it recovers, reacquires leases, then drops again. Each cycle lasts seconds to minutes. Application queries see intermittent timeouts, ambiguous results, and latency spikes that correlate with the node&amp;rsquo;s oscillation. The DB Console shows the node flapping between live and not-live states.&lt;/p&gt;&#10;&lt;p&gt;This is the memory pressure to GC thrashing to Raft liveness failure cascade. The Go runtime heap grows until garbage collection pauses become long enough to prevent the node from renewing its liveness heartbeat. Once the heartbeat interval lapses, the cluster marks the node dead and moves its leases. When GC completes and memory is freed, the node recovers and reacquires leases, restarting the cycle.&lt;/p&gt;</description></item><item><title>CockroachDB out of memory: sys_rss, --cache, --max-sql-memory, and the OOM killer</title><link>https://www.netdata.cloud/guides/cockroachdb/cockroachdb-out-of-memory-oom/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/cockroachdb/cockroachdb-out-of-memory-oom/</guid><description>&lt;p&gt;A CockroachDB node disappears. No graceful shutdown, no drain sequence, no error in the SQL layer. The process is gone, and &lt;code&gt;dmesg&lt;/code&gt; shows the kernel OOM killer selected it. Or in Kubernetes, the pod restarts with reason &lt;code&gt;OOMKilled&lt;/code&gt;.&lt;/p&gt;&#10;&lt;p&gt;The root cause is almost always a mismatch between what CockroachDB thinks it can allocate and what the container or host actually allows. CockroachDB partitions its memory into two manually-sized pools: the Pebble block cache (&lt;code&gt;--cache&lt;/code&gt;) and the SQL execution budget (&lt;code&gt;--max-sql-memory&lt;/code&gt;). The Go garbage collector only manages the Go heap. CGo allocations, primarily the Pebble block cache and memtables, are manually managed. When the sum of these pools plus runtime overhead exceeds the container or host limit, the OOM killer intervenes.&lt;/p&gt;</description></item><item><title>Docker container high memory usage: how to diagnose it</title><link>https://www.netdata.cloud/guides/docker/docker-container-high-memory-usage/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/docker/docker-container-high-memory-usage/</guid><description>&lt;p&gt;Your container is sitting at 90% of its memory limit but has not been OOMKilled. Or it is being killed repeatedly and you cannot tell whether the limit is too low or the application is leaking. docker stats shows a single percentage, but that number does not reveal the breakdown between reclaimable page cache and anonymous memory that the kernel cannot reclaim. To diagnose this correctly, you need to decompose cgroup memory.stat, map it to your runtime&amp;rsquo;s actual allocations, and decide whether the problem is cache pressure, a runtime mismatch, or a true leak.&lt;/p&gt;</description></item><item><title>Docker container memory leak: how to find one and prove it</title><link>https://www.netdata.cloud/guides/docker/docker-container-memory-leak/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/docker/docker-container-memory-leak/</guid><description>&lt;p&gt;Memory that only ever climbs is easy to spot. The harder problem is proving whether the growth is a leak, unbounded caching, or a limit set below the working set. During an incident, operators need to decide in minutes whether to page an on-call developer or bump a cgroup limit. This guide shows how to use cgroup memory.stat, process-level RSS, and container restart patterns to build a defensible diagnosis. You will be able to separate anonymous memory growth from &lt;a href="https://www.netdata.cloud/guides/docker/docker-memory-usage-explained/"&gt;reclaimable cache&lt;/a&gt;, identify whether the leak lives in application heap or runtime overhead, and present evidence that justifies either a code fix or a capacity change.&lt;/p&gt;</description></item><item><title>Docker exit code 137: OOMKilled or SIGKILL?</title><link>https://www.netdata.cloud/guides/docker/docker-exit-code-137/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/docker/docker-exit-code-137/</guid><description>&lt;p&gt;A container exits with code 137. Docker restarts it, or it stays down, and you need to know why. The number itself only tells you that the process received SIGKILL. What matters for your next step is whether the kernel&amp;rsquo;s cgroup OOM killer fired because the container exceeded its memory limit, or whether an external actor sent the signal. The remediation for an undersized memory limit is completely different from fixing a misconfigured stop timeout or an orchestrator sending a premature kill. This guide shows how to classify the cause in under a minute using only the Docker CLI and cgroup files.&lt;/p&gt;</description></item><item><title>Docker JVM memory tuning: heap, off-heap, and the cgroup mismatch</title><link>https://www.netdata.cloud/guides/docker/docker-jvm-memory-tuning/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/docker/docker-jvm-memory-tuning/</guid><description>&lt;p&gt;Your Java container is &lt;a href="https://www.netdata.cloud/guides/docker/docker-oomkilled/"&gt;OOMKilled&lt;/a&gt; at 02:00. Heap usage is 60%. Docker reports exit code 137 and &lt;code&gt;OOMKilled: true&lt;/code&gt;. The JVM never threw an &lt;code&gt;OutOfMemoryError&lt;/code&gt;.&lt;/p&gt;&#10;&lt;p&gt;In a container, the kernel enforces memory limits through cgroups, but the JVM heap is only one component of process RSS. Off-heap memory, metaspace, thread stacks, direct byte buffers, and GC overhead all count against the same cgroup limit. When total RSS crosses that limit, the kernel kills the container without a JVM-level error. In some JDK and kernel combinations, the JVM fails to detect the cgroup limit entirely and sizes the heap against host RAM, which guarantees an OOM kill.&lt;/p&gt;</description></item><item><title>Docker memory limits: how to set them and what happens when they hit</title><link>https://www.netdata.cloud/guides/docker/docker-memory-limits/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/docker/docker-memory-limits/</guid><description>&lt;p&gt;A container without a memory limit can consume all available RAM, force the kernel to reclaim page cache, push the system into swap, and eventually trigger a host-level OOM kill that takes down the Docker daemon or other critical processes. Setting &lt;code&gt;--memory&lt;/code&gt; is not enough: limits can be silently ignored, misread by runtimes, or masked by swap behavior that turns a clean failure into a slow crawl.&lt;/p&gt;&#10;&lt;p&gt;Exit code 137 can mean an OOM kill or an external SIGKILL. The difference matters because the fix for an undersized limit is not the same as the fix for a runtime sizing its heap from host memory instead of the cgroup limit.&lt;/p&gt;</description></item><item><title>Docker memory usage explained: anonymous, file, slab, and what counts</title><link>https://www.netdata.cloud/guides/docker/docker-memory-usage-explained/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/docker/docker-memory-usage-explained/</guid><description>&lt;p&gt;You look at &lt;code&gt;docker stats&lt;/code&gt;, see a container sitting at 1.2 GB of a 1.5 GB limit, and assume it is about to explode. It might be fine. That total includes page cache the kernel will reclaim the second another process asks for memory. Meanwhile, a different container reports 600 MB with no limit set, yet its anonymous memory grows 50 MB per hour and will force a host-level OOM kill before lunch.&lt;/p&gt;</description></item><item><title>Docker OOMKilled: causes, detection, and prevention</title><link>https://www.netdata.cloud/guides/docker/docker-oomkilled/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/docker/docker-oomkilled/</guid><description>&lt;p&gt;A container exits with code 137 and restarts. The application loses in-memory state. Dependent services start failing. The restart loop begins. This is the OOMKilled pattern, and it is one of the most common and most misdiagnosed failure modes in Docker environments.&lt;/p&gt;&#10;&lt;p&gt;This article covers how to confirm an OOM kill, distinguish it from an external SIGKILL, understand why it happened, and prevent recurrence. It also covers the JVM-in-container memory mismatch, which is responsible for a large share of OOM kills in Java workloads.&lt;/p&gt;</description></item><item><title>Elasticsearch cluster state too large: field count, index count, and per-node heap</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-cluster-state-too-large/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-cluster-state-too-large/</guid><description>&lt;p&gt;Every node holds a copy of the cluster state in heap. When it grows large, the cost is paid everywhere: 200 MB of state consumes 200 MB on every node, and the elected master burns additional CPU and heap serializing and publishing updates. Symptoms show up indirectly: the master feels sluggish, pending tasks queue for minutes, heap pressure climbs on nodes that should be idle, and master elections stall indexing and shard allocation. The usual drivers are too many indices from per-minute or per-hour time-series patterns; a mapping explosion from uncontrolled dynamic fields; excessive aliases; or churn from frequent template and setting changes. Raw size measured via &lt;code&gt;/_cluster/state&lt;/code&gt; is a rough proxy. The indicators that matter are field-count growth, cluster state version churn, pending-task age, and master node heap pressure.&lt;/p&gt;</description></item><item><title>Elasticsearch coordinating node overload: aggregation merge, heap spikes, and 429s</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-coordinating-node-overload/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-coordinating-node-overload/</guid><description>&lt;p&gt;HTTP 429 or 503 responses appear on search requests while data nodes look healthy. Heap spikes on one node while others stay flat. The slow log shows heavy aggregation queries. That node is the coordinator, and it is running out of heap during the reduce phase.&lt;/p&gt;&#10;&lt;p&gt;Every node can act as a coordinating node. For each search, the coordinator broadcasts the query to relevant shards, collects partial results, and merges them. Aggregations compute locally per shard and reduce in memory on the coordinator. High-cardinality terms aggregations, deep pagination, or large fetch sizes force the coordinator to hold massive intermediate structures in heap. If the estimate exceeds the circuit breaker limit, the request is rejected. If the breaker is too slow, the node may suffer long GC pauses or disconnect from the cluster.&lt;/p&gt;</description></item><item><title>Elasticsearch fielddata circuit breaker tripped: text-field aggregations and the keyword fix</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-fielddata-circuit-breaker-tripped/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-fielddata-circuit-breaker-tripped/</guid><description>&lt;p&gt;Queries return &lt;code&gt;CircuitBreakingException: [fielddata] Data too large...&lt;/code&gt; and HTTP 429s while JVM heap on one or more data nodes climbs toward the breaker limit. The node rejects queries to protect itself before OOM. This almost always means a query is aggregating, sorting, or scripting against an analyzed &lt;code&gt;text&lt;/code&gt; field that lacks a &lt;code&gt;keyword&lt;/code&gt; sub-field, forcing Elasticsearch to load an expensive fielddata cache into heap.&lt;/p&gt;&#10;&lt;p&gt;Analyzed &lt;code&gt;text&lt;/code&gt; fields are tokenized and optimized for full-text search, not columnar operations such as &lt;code&gt;terms&lt;/code&gt; aggregations or sorting. Running these against a raw &lt;code&gt;text&lt;/code&gt; field forces Elasticsearch to uninvert the inverted index per segment into an in-memory structure called fielddata. This cache consumes JVM heap proportional to unique term cardinality. On large indices it approaches the fielddata circuit breaker limit, which defaults to 40% of the JVM heap.&lt;/p&gt;</description></item><item><title>Elasticsearch heap pressure death spiral: GC, node removal, and the cascade</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-heap-pressure-death-spiral/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-heap-pressure-death-spiral/</guid><description>&lt;p&gt;When a node drops from &lt;code&gt;_cat/nodes&lt;/code&gt; and network tests pass, check its JVM GC logs. Stop-the-world pauses over 10 seconds cause the master to remove the node. Survivors then absorb recovery traffic, their heap climbs, and they begin missing fault-detection checks too. This feedback loop is the heap pressure death spiral. It masquerades as network instability because operators check connectivity while the real problem is memory saturation.&lt;/p&gt;&#10;&lt;h2 id="what-this-means"&gt;What this means&lt;/h2&gt;&#10;&lt;p&gt;By default, Elasticsearch fault detection removes a node after roughly 10-15 seconds of sustained unresponsiveness. A stop-the-world GC pause that prevents the node from responding to fault-detection checks triggers this removal. A hard TCP disconnect removes the node immediately.&lt;/p&gt;</description></item><item><title>Elasticsearch JVM heap usage high: reading the sawtooth and the post-GC floor</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-jvm-heap-high/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-jvm-heap-high/</guid><description>&lt;p&gt;Your Elasticsearch alert fires: &lt;code&gt;jvm.mem.heap_used_percent&lt;/code&gt; has crossed 75 percent and is holding there. You pull up the graph and see a jagged sawtooth climbing toward the ceiling. The first instinct is to add heap or restart the node. Both are usually wrong.&lt;/p&gt;&#10;&lt;p&gt;The sawtooth is normal. Elasticsearch runs on the JVM with a young generation that fills with short-lived objects and empties on young garbage collections. The peak of the tooth is noise. The signal that matters is the post-GC floor: the minimum heap used immediately after a collection. In a healthy node, the floor stays between roughly 30 and 50 percent of max heap, and young GC dominates. When the floor trends upward, old generation objects are accumulating. Old GC pauses stop the world, and once a pause exceeds the cluster fault detection timeout, the master removes the node and triggers shard reallocation. That reallocation places more heap pressure on the survivors, beginning a death spiral.&lt;/p&gt;</description></item><item><title>Elasticsearch long GC pauses: old-generation stop-the-world and node drops</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-old-gc-long-pauses/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-old-gc-long-pauses/</guid><description>&lt;p&gt;In Elasticsearch 8.x, nodes can drop out of the cluster without logging errors. The master logs &lt;code&gt;node-left&lt;/code&gt; with reason &lt;code&gt;disconnected&lt;/code&gt;, while the departed node shows no ERROR entries because its JVM was frozen in an old-generation stop-the-world GC pause. A single pause longer than 10 seconds fails a fault-detection check; roughly 30 seconds of total unresponsiveness triggers removal. Once the master reallocates shards, remaining nodes face additional heap pressure and the cascade continues.&lt;/p&gt;</description></item><item><title>Elasticsearch mapping explosion: dynamic mapping, cluster state bloat, and master pressure</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-mapping-explosion-dynamic-mapping/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-mapping-explosion-dynamic-mapping/</guid><description>&lt;p&gt;When an Elasticsearch cluster ingesting unstructured JSON starts accumulating thousands of mapped fields per index, every node pays for it in heap. Mappings are part of the cluster state, which the elected master serializes and publishes to every node on each change. An index mapping with tens of thousands of fields means a larger cluster state blob resident in every node&amp;rsquo;s JVM heap, longer publication times, and mounting pressure on the master.&lt;/p&gt;</description></item><item><title>Elasticsearch master instability: frequent elections and metadata overload</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-master-instability-flapping/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-master-instability-flapping/</guid><description>&lt;p&gt;Index creation requests time out. &lt;code&gt;_cluster/health&lt;/code&gt; hangs or returns timeouts. The node listed by &lt;code&gt;_cat/master&lt;/code&gt; changes every few minutes outside planned maintenance. Shard allocation stalls, and new indices stay red or unassigned even though all data nodes are reachable. These symptoms indicate a master node that cannot keep up with cluster state updates, triggering repeated elections and leaving the cluster without stable coordination.&lt;/p&gt;&#10;&lt;p&gt;This is metadata overload. The elected master maintains the cluster state: a heap-resident data structure describing every index, shard, mapping, alias, pipeline, and node. On every change, the master serializes and publishes the state to all nodes. Updates are processed serially, so any delay in serialization, heap allocation, or node acknowledgment blocks subsequent metadata operations. When metadata churn is high or the state is oversized, the master falls behind, pending tasks accumulate, and if the master misses enough heartbeat checks, remaining master-eligible nodes trigger a new election. Until a stable master converges, writes, allocations, and administrative operations stall.&lt;/p&gt;</description></item><item><title>Elasticsearch monitoring checklist: the signals every production cluster needs</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-monitoring-checklist/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-monitoring-checklist/</guid><description>&lt;p&gt;Elasticsearch failures cascade. A long GC pause on one node causes it to miss fault detection checks; the master removes it. Shards relocate to survivors, increasing heap pressure and thread pool load. If disk is near the high watermark, relocation I/O pushes other nodes toward flood stage, which sets indices to read-only and blocks writes. By the time &lt;code&gt;GET /_cluster/health&lt;/code&gt; returns red, the leading indicators fired minutes ago.&lt;/p&gt;&#10;&lt;p&gt;Cluster health is a lagging indicator. It tells you damage is done, not that it is coming. This checklist organizes signals into four levels. Use it to audit coverage or build instrumentation. Start with survival, but build dashboards that show all four levels so correlations are obvious.&lt;/p&gt;</description></item><item><title>Elasticsearch node left the cluster: fault detection, reallocation, and recovery</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-node-left-cluster/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-node-left-cluster/</guid><description>&lt;p&gt;Your cluster health turned yellow and &lt;code&gt;number_of_nodes&lt;/code&gt; dropped by one. The master logs a &lt;code&gt;NODE_LEFT&lt;/code&gt; event, shards are unassigned, and the remaining nodes absorb extra load. In the next minute, the allocator decides whether to move data. Misread the cause and a transient restart becomes an expensive reallocation storm, or a genuine hardware failure goes unaddressed while replicas rebalance.&lt;/p&gt;&#10;&lt;p&gt;This guide covers how Elasticsearch decides a node is gone, what happens to its shards, and how to recover without deepening the incident.&lt;/p&gt;</description></item><item><title>Elasticsearch node OOM-killed: heap ceiling, page cache, and container limits</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-out-of-memory-oom-killed/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-out-of-memory-oom-killed/</guid><description>&lt;p&gt;An Elasticsearch node leaves the cluster, restarts seconds later via systemd or a supervisor, and is killed again. Kernel logs show the OOM-killer terminated the Java process. &lt;code&gt;heap.percent&lt;/code&gt; often looks reasonable right up until the kill.&lt;/p&gt;&#10;&lt;p&gt;The JVM heap is only one component of resident set size. Off-heap allocations, memory-mapped Lucene segments, and co-located processes all compete for the same memory budget. In containers, the cgroup limit is the hard boundary, not the host&amp;rsquo;s physical RAM.&lt;/p&gt;</description></item><item><title>Fix Elasticsearch CircuitBreakingException Errors</title><link>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-circuitbreakingexception-parent-data-too-large/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/elasticsearch/elasticsearch-circuitbreakingexception-parent-data-too-large/</guid><description>&lt;p&gt;When a search or indexing request returns HTTP 429 with &lt;code&gt;CircuitBreakingException: [parent] Data too large, data for [&amp;lt;http_request&amp;gt;] would be [X], which is larger than the limit of [Y]&lt;/code&gt;, the parent circuit breaker has rejected the operation. This is Elasticsearch protecting the JVM from an out-of-memory kill, not a client-side rate limit.&lt;/p&gt;&#10;&lt;p&gt;Since version 7.0, the parent breaker tracks real memory usage by default. It can trip even when individual child breakers are within limits. The node is under genuine heap pressure. Determine quickly whether the cause is a single abusive query or structural memory exhaustion.&lt;/p&gt;</description></item><item><title>Kafka JVM heap and Full GC pauses: ISR drops, session timeouts, and right-sizing the heap</title><link>https://www.netdata.cloud/guides/kafka/kafka-jvm-heap-full-gc-pauses/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/kafka/kafka-jvm-heap-full-gc-pauses/</guid><description>&lt;p&gt;Sporadic &lt;code&gt;UnderReplicatedPartitions&lt;/code&gt; and ISR shrinks that do not correlate with disk I/O or network faults, combined with consumer rebalances and &lt;code&gt;NotEnoughReplicasException&lt;/code&gt; from producers using &lt;code&gt;acks=all&lt;/code&gt;, point to broker JVM heap pressure. Check broker logs for GC pauses in the Old Generation lasting several seconds.&lt;/p&gt;&#10;&lt;p&gt;Brokers use the JVM heap for metadata, request buffers, and message format conversion. They do not store messages on the heap; the OS page cache handles that. When the heap is misconfigured or under pressure, garbage collection pauses can freeze a broker long enough to trigger ZooKeeper session expirations, follower lag, and cascading availability issues. Full GC pauses exceeding five seconds are the common threshold where these symptoms begin.&lt;/p&gt;</description></item><item><title>Kernel Same-page Merging (KSM) Monitoring</title><link>https://www.netdata.cloud/monitoring-101/ksm-monitoring/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/monitoring-101/ksm-monitoring/</guid><description>&lt;h2 id="kernel-same-page-merging-ksm"&gt;Kernel Same-page Merging (KSM)?&lt;/h2&gt;&#10;&lt;p&gt;Linux kernels store memory in &lt;strong&gt;pages&lt;/strong&gt; which are moved in and out of memory as a single block. On most Linux architectures pages are 4096 bytes. &lt;strong&gt;KSM&lt;/strong&gt; (Kernel Same-page Merging) is a kernel feature that scans memory looking for pages with identical content, and then de-duplicates them. The most common use-case where such duplicate pages occur is on hosts running multiple virtual machines (VMs).&lt;/p&gt;&#10;&lt;p&gt;KSM can greatly reduce the amount of memory used by VMs. When it finds two or more identical pages, it replaces them with a single page that is shared by all VMs that are using it.&lt;/p&gt;</description></item><item><title>Kubernetes eviction cascade: when one node failure takes down the cluster</title><link>https://www.netdata.cloud/guides/kubernetes/kubernetes-eviction-cascade/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/kubernetes/kubernetes-eviction-cascade/</guid><description>&lt;p&gt;You see pods entering Evicted status across multiple nodes. Nodes flap between Ready and MemoryPressure or DiskPressure. The scheduler keeps placing replacements, but the new pods are evicted again before they become ready. Workloads never stabilize, and every remediation attempt seems to make the cluster more volatile.&lt;/p&gt;&#10;&lt;p&gt;This is a node-pressure eviction cascade. It happens when the scheduler&amp;rsquo;s view of capacity diverges from the kubelet&amp;rsquo;s view. One node under pressure evicts pods; those pods land on other nodes that are also overcommitted; those nodes tip into pressure and evict more pods. The result is a cluster-wide feedback loop that looks like a resource shortage but is often a scheduling and configuration problem.&lt;/p&gt;</description></item><item><title>Kubernetes node MemoryPressure: detection, eviction order, and prevention</title><link>https://www.netdata.cloud/guides/kubernetes/kubernetes-node-memory-pressure/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/kubernetes/kubernetes-node-memory-pressure/</guid><description>&lt;p&gt;Before adding RAM, determine whether kubelet is evicting because workloads are genuinely starving or because memory requests are misaligned with reality.&lt;/p&gt;&#10;&lt;h2 id="what-this-means"&gt;What this means&lt;/h2&gt;&#10;&lt;p&gt;Kubelet evaluates &lt;code&gt;memory.available&lt;/code&gt; against an eviction threshold. On Linux the default hard threshold is &lt;code&gt;memory.available &amp;lt; 100Mi&lt;/code&gt;. Kubelet derives this from cgroup stats, not &lt;code&gt;free -m&lt;/code&gt;. It measures working-set memory (RSS plus active file-backed pages) and subtracts that from total capacity. When the threshold is crossed, kubelet sets the node condition &lt;code&gt;MemoryPressure=True&lt;/code&gt; and adds the taint &lt;code&gt;node.kubernetes.io/memory-pressure:NoSchedule&lt;/code&gt;. New pods are blocked from scheduling until the condition clears.&lt;/p&gt;</description></item><item><title>Kubernetes pod CrashLoopBackOff: causes, diagnosis, and fixes</title><link>https://www.netdata.cloud/guides/kubernetes/kubernetes-pod-crashloopbackoff/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/kubernetes/kubernetes-pod-crashloopbackoff/</guid><description>&lt;p&gt;CrashLoopBackOff means a container in a Pod has terminated after starting, and the kubelet is delaying the next restart with exponential backoff. The status describes behavior, not root cause. Underlying failures include application panics, OOM kills, misconfigured liveness probes, missing secrets, or node-level resource pressure.&lt;/p&gt;&#10;&lt;p&gt;Use pod status, previous container logs, node conditions, and kubelet events to narrow the cause. Monitor restart rate, node pressure, and probe failures to catch loops before they degrade capacity.&lt;/p&gt;</description></item><item><title>Kubernetes pod Evicted: detection, root cause, and prevention</title><link>https://www.netdata.cloud/guides/kubernetes/kubernetes-pod-evicted/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/kubernetes/kubernetes-pod-evicted/</guid><description>&lt;p&gt;Pods with status &lt;code&gt;Evicted&lt;/code&gt; are not application crashes. They are the kubelet&amp;rsquo;s emergency response to node-level resource pressure. When memory, disk, inodes, or PIDs approach exhaustion, the kubelet terminates pods to reclaim resources and protect node availability. The pod phase changes to &lt;code&gt;Failed&lt;/code&gt; with reason &lt;code&gt;Evicted&lt;/code&gt;, and the node reports conditions such as &lt;code&gt;MemoryPressure&lt;/code&gt; or &lt;code&gt;DiskPressure&lt;/code&gt;.&lt;/p&gt;&#10;&lt;p&gt;This guide covers node-pressure eviction triggered by the kubelet, not voluntary disruption from &lt;code&gt;kubectl drain&lt;/code&gt; or PodDisruptionBudget enforcement.&lt;/p&gt;</description></item><item><title>Kubernetes pod OOMKilled: cgroup limits, evictions, and fixes</title><link>https://www.netdata.cloud/guides/kubernetes/kubernetes-pod-oomkilled/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/kubernetes/kubernetes-pod-oomkilled/</guid><description>&lt;p&gt;A pod status of &lt;code&gt;OOMKilled&lt;/code&gt; means the container restarted after the kernel sent SIGKILL because it could not satisfy a memory allocation. There is no graceful shutdown.&lt;/p&gt;&#10;&lt;p&gt;Distinguish whether the kill happened at the container cgroup level (a limit you set) or at the node level (a system-wide shortage). Then separate kernel OOM kills from kubelet evictions, identify the correct fix, and prevent recurrence without guessing at memory limits.&lt;/p&gt;&#10;&lt;h2 id="what-this-means"&gt;What this means&lt;/h2&gt;&#10;&lt;p&gt;&lt;code&gt;OOMKilled&lt;/code&gt; means the Linux OOM killer selected a process in your container and terminated it with SIGKILL (signal 9). Exit code 137 (128 + 9) often maps to &lt;code&gt;OOMKilled&lt;/code&gt;, but the code alone is ambiguous: it can also result from a manual &lt;code&gt;docker kill&lt;/code&gt; or an escalated graceful termination. Verify the pod status &lt;code&gt;Reason&lt;/code&gt; field.&lt;/p&gt;</description></item><item><title>MongoDB Application Thread Evictions: How To Fix</title><link>https://www.netdata.cloud/guides/mongodb/mongodb-application-thread-evictions/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/mongodb/mongodb-application-thread-evictions/</guid><description>&lt;p&gt;Query p99 latency doubles or triples, but &lt;code&gt;iostat&lt;/code&gt; is not saturated and the slow query log shows no single offender. The signal is in &lt;code&gt;db.serverStatus().wiredTiger.cache&lt;/code&gt;: &lt;code&gt;pages evicted by application threads&lt;/code&gt; has moved from zero to a sustained nonzero rate.&lt;/p&gt;&#10;&lt;p&gt;This metric marks the moment when WiredTiger&amp;rsquo;s dedicated eviction workers fall behind and application threads are drafted to do the work. Any sustained nonzero rate is abnormal. Once application threads evict, they perform page reconciliation and disk I/O inline with the request handler thread, directly inflating user-visible latency. The companion counter &lt;code&gt;pages selected for eviction unable to be evicted&lt;/code&gt; means eviction is stalled and the cache is effectively frozen.&lt;/p&gt;</description></item><item><title>MongoDB cache too small: sizing the WiredTiger cache for your working set</title><link>https://www.netdata.cloud/guides/mongodb/mongodb-cache-undersized-working-set/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/mongodb/mongodb-cache-undersized-working-set/</guid><description>&lt;p&gt;When MongoDB latency doubles and disk read IOPS climb, operators usually check indexes and the query planner first. If &lt;code&gt;db.currentOp()&lt;/code&gt; shows no runaway query and the slow query log is quiet, the culprit is often the WiredTiger cache.&lt;/p&gt;&#10;&lt;p&gt;WiredTiger maintains its own in-memory cache of uncompressed B-tree pages, separate from the OS page cache. MongoDB defaults the cache to &lt;code&gt;max(0.5 * (RAM - 1 GB), 256 MB)&lt;/code&gt;. That default works for a single mongod on a dedicated host, but it breaks down in containers, multi-tenant deployments, and during organic data growth. Once the working set exceeds the cache, reads fault to disk, pages are decompressed, and eviction threads compete with application threads for CPU.&lt;/p&gt;</description></item><item><title>MongoDB connection storm spiral: reconnection floods after an election or deploy</title><link>https://www.netdata.cloud/guides/mongodb/mongodb-connection-storm-spiral/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/mongodb/mongodb-connection-storm-spiral/</guid><description>&lt;p&gt;Connection count on a primary jumps from 200 to 4,000 in under a minute. Resident memory climbs, query latencies double, and application logs fill with timeout errors. The slow query log shows nothing unusual. Individual queries are not the problem. The database is drowning in threads.&lt;/p&gt;&#10;&lt;p&gt;This is a connection storm spiral. A trigger event, usually a replica set election, application deploy, or network blip, invalidates existing connections across your application fleet. Every driver reconnects at once. Each new connection costs MongoDB a dedicated thread and roughly 1 MB of stack memory. The resulting RSS spike and ticket contention slow down operations already in flight, causing more timeouts, which drives even more reconnections. The feedback loop ends in OOM kill or unresponsiveness.&lt;/p&gt;</description></item><item><title>MongoDB exceeded memory limit for $group — aggregation spills and allowDiskUse</title><link>https://www.netdata.cloud/guides/mongodb/mongodb-exceeded-memory-limit-group-sort/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/mongodb/mongodb-exceeded-memory-limit-group-sort/</guid><description>&lt;p&gt;Application logs show error 16945, or an aggregation pipeline slows by an order of magnitude. In MongoDB, every aggregation stage not backed by an index is limited to 100 megabytes of RAM. When a stage exceeds this limit and disk spilling is not enabled, the operation fails immediately. If spilling is enabled, MongoDB writes temporary files to disk, which keeps the pipeline alive but adds unpredictable latency and extra I/O load.&lt;/p&gt;</description></item><item><title>MongoDB OOM-killed by the kernel: RSS, cache sizing, and oom_score_adj</title><link>https://www.netdata.cloud/guides/mongodb/mongodb-oom-killed/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/mongodb/mongodb-oom-killed/</guid><description>&lt;p&gt;You find &lt;code&gt;mongod&lt;/code&gt; gone. The replica set has no primary. Applications time out. MongoDB logs show no graceful shutdown. Instead, &lt;code&gt;dmesg&lt;/code&gt; shows &lt;code&gt;Out of memory: Killed process 12345 (mongod)&lt;/code&gt;. The Linux OOM killer has reaped the process. MongoDB is a frequent target because its resident set size is usually the largest on the host.&lt;/p&gt;&#10;&lt;p&gt;An OOM kill is not a MongoDB bug. It is the kernel freeing RAM by terminating the highest-scoring process. mongod&amp;rsquo;s RSS is dominated by the WiredTiger cache, plus roughly 1 MB per connection, plus roughly 500 MB to 1 GB of internal overhead for indexes, session buffers, and stack. When that sum comes within 1 GB of total RAM, the node is in the danger zone. The kill is abrupt: no stepdown, no replica set coordination, and after restart the cache must warm again.&lt;/p&gt;</description></item><item><title>MongoDB page faults high: working set exceeding memory after warmup</title><link>https://www.netdata.cloud/guides/mongodb/mongodb-page-faults-high/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/mongodb/mongodb-page-faults-high/</guid><description>&lt;p&gt;Hard page faults long after startup mean the active data set exceeds resident memory. On Linux, &lt;code&gt;extra_info.page_faults&lt;/code&gt; counts major faults: the OS read data from disk because the page was missing from both the WiredTiger cache and the OS page cache. A brief spike after restart is normal during warmup, but sustained faults mean the working set does not fit. On EBS gp3, 50 faults per second can degrade latency. On NVMe, hundreds per second may be tolerable, but neither is free. Confirm the cause, distinguish warmup from pressure, and reduce the fault rate without guessing.&lt;/p&gt;</description></item><item><title>MongoDB RSS growing without cache growth: leaks, threads, and tcmalloc fragmentation</title><link>https://www.netdata.cloud/guides/mongodb/mongodb-memory-rss-growing/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/mongodb/mongodb-memory-rss-growing/</guid><description>&lt;p&gt;&lt;code&gt;db.serverStatus().mem.resident&lt;/code&gt; climbs while WiredTiger cache utilization stays flat and the host is not swapping. Virtual memory is larger than RSS by design and is not an alert target. Only RSS reflects physical memory pressure. When RSS grows without cache growth, the problem lives outside the storage engine.&lt;/p&gt;&#10;&lt;p&gt;This pattern points to one of three areas: tcmalloc heap retention and fragmentation, per-connection thread stack accumulation, or unbounded internal allocations from cursors, plan caches, or aggregation pipelines. Each connection reserves roughly 1 MB of stack space, so a connection storm can add gigabytes of RSS in minutes. TCMalloc caches freed memory in per-thread or per-CPU arenas, which inflates RSS independently of the WiredTiger cache.&lt;/p&gt;</description></item><item><title>MySQL InnoDB buffer pool hit ratio collapse: the cliff edge</title><link>https://www.netdata.cloud/guides/mysql/mysql-buffer-pool-hit-ratio-collapse/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/mysql/mysql-buffer-pool-hit-ratio-collapse/</guid><description>&lt;p&gt;Your OLTP queries were running in single-digit milliseconds. Now every query is taking seconds, the disk subsystem is saturated, &lt;code&gt;Threads_running&lt;/code&gt; is climbing toward &lt;code&gt;max_connections&lt;/code&gt;, and the buffer pool hit ratio, which sat at 99.9% for months, just fell through 95% and keeps dropping.&lt;/p&gt;&#10;&lt;p&gt;This is the InnoDB buffer pool cliff edge. When the working set exceeds the buffer pool, pages are evicted before they can be reused. Every miss becomes a physical disk read. The degradation is non-linear: 99.9% to 99% is a slow bleed, 99% to 95% is rapid, and below 95% disk saturation, uniform latency inflation, and connection exhaustion turn a capacity problem into an availability incident.&lt;/p&gt;</description></item><item><title>MySQL innodb_buffer_pool_size tuning: 60-80% of RAM and when that breaks</title><link>https://www.netdata.cloud/guides/mysql/mysql-buffer-pool-sizing/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/mysql/mysql-buffer-pool-sizing/</guid><description>&lt;p&gt;The 60-80% rule works for a bare-metal host running only MySQL. In containers, shared hardware, and high-connection-count environments, it is a hazard. Size the pool against aligned allocation, cgroup limits, per-connection memory, and OS headroom. Do not size it as a percentage of total RAM.&lt;/p&gt;&#10;&lt;h2 id="what-the-buffer-pool-costs"&gt;What the buffer pool costs&lt;/h2&gt;&#10;&lt;p&gt;InnoDB caches data and index pages in a fixed-size pool. Every read and write touches it. When the working set fits entirely in memory, queries avoid disk. When it does not, InnoDB evicts pages and disk reads dominate latency.&lt;/p&gt;</description></item><item><title>MySQL Innodb_buffer_pool_wait_free &gt; 0: buffer pool memory pressure</title><link>https://www.netdata.cloud/guides/mysql/mysql-buffer-pool-wait-free/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/mysql/mysql-buffer-pool-wait-free/</guid><description>&lt;p&gt;&lt;code&gt;Innodb_buffer_pool_wait_free&lt;/code&gt; increments when InnoDB must synchronously flush dirty pages to make room for new reads. A sustained nonzero rate means queries are waiting on disk writes before they can proceed. Unlike the buffer pool hit ratio, which can stay above 99% while the system stalls, &lt;code&gt;wait_free&lt;/code&gt; confirms the buffer pool is operating at its limit.&lt;/p&gt;&#10;&lt;p&gt;This is the Buffer Pool Cliff pattern: once the working set exceeds available clean pages, performance degrades non-linearly, disk I/O saturates, and threads pile up.&lt;/p&gt;</description></item><item><title>MySQL OOM-killed: buffer pool, per-connection buffers, and the kernel killer</title><link>https://www.netdata.cloud/guides/mysql/mysql-out-of-memory-oom-killed/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/mysql/mysql-out-of-memory-oom-killed/</guid><description>&lt;p&gt;MySQL disappears. Application logs fill with connection timeouts. &lt;code&gt;Uptime&lt;/code&gt; resets to near zero. In the kernel log: &lt;code&gt;oom-kill: task mysqld&lt;/code&gt; and its anonymous RSS. Orchestrators mark the pod &lt;code&gt;OOMKilled&lt;/code&gt; (exit code 137). To the application, this is a crash. To the kernel, mysqld was the largest memory consumer and the system ran out.&lt;/p&gt;&#10;&lt;p&gt;OOM kills are gradual, then catastrophic. Memory pressure builds as connections open, temp tables materialize, and dirty pages accumulate. The buffer pool is the obvious consumer, but the killer often enters through the back door: a connection burst multiplies per-thread buffers, a container limit sits too close to the buffer pool size, or Transparent Huge Pages block reclamation. Map all allocators to prevent recurrence.&lt;/p&gt;</description></item><item><title>PostgreSQL Out Of Memory: OOM Killer, shared_buffers &amp; work_mem</title><link>https://www.netdata.cloud/guides/postgres/postgres-out-of-memory/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/postgres/postgres-out-of-memory/</guid><description>&lt;p&gt;Your PostgreSQL primary restarts without warning, or individual backends vanish from the process list. The kernel log shows &lt;code&gt;Out of Memory: Killed process 12345 (postgres)&lt;/code&gt;. Existing connections may survive, but new connections fail until the postmaster recovers. This is a memory accounting mismatch between Linux overcommit, PostgreSQL shared and private memory allocation, and how you size &lt;code&gt;shared_buffers&lt;/code&gt; and &lt;code&gt;work_mem&lt;/code&gt;.&lt;/p&gt;&#10;&lt;p&gt;Inside Kubernetes or containers, the symptom is identical but the mechanism differs: cgroup v2 &lt;code&gt;memory.max&lt;/code&gt; triggers an immediate SIGKILL with no ENOMEM grace period. In both cases, stop guessing at memory limits and start budgeting.&lt;/p&gt;</description></item><item><title>RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster</title><link>https://www.netdata.cloud/guides/rabbitmq/rabbitmq-memory-resource-limit-alarm/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/rabbitmq/rabbitmq-memory-resource-limit-alarm/</guid><description>&lt;p&gt;Your RabbitMQ cluster just stopped accepting messages. Publishers are connected but nothing flows. The broker log shows:&lt;/p&gt;&#10;&lt;pre tabindex="0"&gt;&lt;code&gt;memory resource limit alarm set on node rabbit@node1.&#10;*** Publishers will be blocked until this alarm clears ***&#10;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;One node crossed its memory high watermark, and RabbitMQ responded by blocking every publisher on every node in the cluster. There is no gradual degradation: the transition from &amp;ldquo;fine&amp;rdquo; to &amp;ldquo;all ingestion halted&amp;rdquo; is a single threshold crossing.&lt;/p&gt;</description></item><item><title>RabbitMQ memory usage climbing toward the watermark: mem_used vs mem_limit</title><link>https://www.netdata.cloud/guides/rabbitmq/rabbitmq-memory-usage-climbing/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/rabbitmq/rabbitmq-memory-usage-climbing/</guid><description>&lt;p&gt;The memory alarm has not fired yet. Publishers are still running. But &lt;code&gt;mem_used&lt;/code&gt; has been creeping up for hours and &lt;code&gt;mem_used / mem_limit&lt;/code&gt; just crossed 0.7. This is the last quiet window you get: RabbitMQ&amp;rsquo;s memory failure mode is a cliff edge. When &lt;code&gt;mem_used&lt;/code&gt; reaches &lt;code&gt;mem_limit&lt;/code&gt;, the alarm fires and every publisher on every node in the cluster is blocked at once.&lt;/p&gt;&#10;&lt;p&gt;Two mistakes are common here. The first is treating &lt;code&gt;mem_limit&lt;/code&gt; as raw RAM you should multiply by the watermark again. It is not; the watermark is already baked in. The second is dismissing a rising ratio because the alarm has not fired. By the time it fires, you have zero recovery headroom.&lt;/p&gt;</description></item><item><title>Redis Can't save in background: fork: Cannot allocate memory - diagnosis and fix</title><link>https://www.netdata.cloud/guides/redis/redis-cant-save-in-background-fork/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/redis/redis-cant-save-in-background-fork/</guid><description>&lt;p&gt;Redis logs &lt;code&gt;Can't save in background: fork: Cannot allocate memory&lt;/code&gt;. &lt;code&gt;free -h&lt;/code&gt; shows plenty of free RAM, yet &lt;code&gt;BGSAVE&lt;/code&gt; or &lt;code&gt;BGREWRITEAOF&lt;/code&gt; fails. If &lt;code&gt;stop-writes-on-bgsave-error&lt;/code&gt; is &lt;code&gt;yes&lt;/code&gt; (default), writes fail too. The gap between free RAM and fork failure is the key.&lt;/p&gt;&#10;&lt;p&gt;This is not a simple OOM. It is a kernel commit charge failure. Linux &lt;code&gt;fork()&lt;/code&gt; must account for the worst case where every copy-on-write page is modified. With &lt;code&gt;vm.overcommit_memory=0&lt;/code&gt; (the default), the kernel enforces a heuristic commit limit. When Redis RSS is large, that limit blocks &lt;code&gt;fork()&lt;/code&gt; even with free physical memory. The fix is usually one sysctl, but THP, container limits, and actual RAM headroom determine whether it holds.&lt;/p&gt;</description></item><item><title>Redis client output buffer overflow: slow consumers and client-output-buffer-limit</title><link>https://www.netdata.cloud/guides/redis/redis-client-output-buffer-limit/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/redis/redis-client-output-buffer-limit/</guid><description>&lt;p&gt;Redis accepts commands faster than some clients can read the responses. When a client&amp;rsquo;s read loop stalls, Redis keeps writing into that client&amp;rsquo;s output buffer because there is no backpressure mechanism for normal clients. The buffer grows on the main heap, and that memory counts against &lt;code&gt;maxmemory&lt;/code&gt;. Under the right conditions, a single slow consumer can push the server into eviction, OOM rejection, or a crash.&lt;/p&gt;&#10;&lt;p&gt;The default &lt;code&gt;client-output-buffer-limit normal 0 0 0&lt;/code&gt; means unlimited. No hard limit, no soft limit, no timeout. This is a footgun that ships with every Redis instance. Pub/Sub and replica clients get defaults, but normal clients, the vast majority of connections, get nothing.&lt;/p&gt;</description></item><item><title>Redis connected_clients climbing: connection leak detection</title><link>https://www.netdata.cloud/guides/redis/redis-connected-clients-climbing/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/redis/redis-connected-clients-climbing/</guid><description>&lt;p&gt;A sustained climb in &lt;code&gt;connected_clients&lt;/code&gt; over hours or days while application traffic is flat is a classic Redis connection leak. Each connection costs roughly 4-8 KB of server-side memory. 10,000 leaked connections consume ~40-80 MB independent of your dataset. If the instance is near &lt;code&gt;maxmemory&lt;/code&gt;, that overhead can push Redis into eviction or OOM territory.&lt;/p&gt;&#10;&lt;p&gt;The default &lt;code&gt;timeout&lt;/code&gt; is 0, so idle connections are never closed. Missing &lt;code&gt;close()&lt;/code&gt; calls, connection pool misconfiguration, or unsubscribed pub/sub listeners all accumulate forever.&lt;/p&gt;</description></item><item><title>Redis eviction policy tuning: allkeys-lru vs volatile-ttl vs noeviction</title><link>https://www.netdata.cloud/guides/redis/redis-eviction-policy-tuning/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/redis/redis-eviction-policy-tuning/</guid><description>&lt;p&gt;When Redis reaches &lt;code&gt;maxmemory&lt;/code&gt;, it must either reject new writes or delete existing keys. The &lt;code&gt;maxmemory-policy&lt;/code&gt; directive decides which path it takes, yet many production instances run with a policy that mismatches the workload. A cache running &lt;code&gt;noeviction&lt;/code&gt; returns OOM errors to clients. A database running &lt;code&gt;allkeys-lru&lt;/code&gt; silently deletes committed data. A session store running &lt;code&gt;volatile-ttl&lt;/code&gt; suddenly rejects writes the moment an application bug omits a TTL.&lt;/p&gt;&#10;&lt;h2 id="what-it-is-and-why-it-matters"&gt;What it is and why it matters&lt;/h2&gt;&#10;&lt;p&gt;Redis stores data in memory. The &lt;code&gt;maxmemory&lt;/code&gt; directive sets the ceiling; when a write would push &lt;code&gt;used_memory&lt;/code&gt; over it, Redis evaluates &lt;code&gt;maxmemory-policy&lt;/code&gt; before executing the command. Eviction runs synchronously in the command path, consuming main-thread CPU and adding latency to the triggering write.&lt;/p&gt;</description></item><item><title>Redis fork/COW memory storm: why persistence doubles RSS and OOM-kills the box</title><link>https://www.netdata.cloud/guides/redis/redis-fork-cow-storm/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/redis/redis-fork-cow-storm/</guid><description>&lt;p&gt;Redis disappeared from your container with only an &lt;code&gt;OOMKilled&lt;/code&gt; status and a metrics gap that aligns with an RDB snapshot or AOF rewrite. The dataset was under its memory limit moments ago, but during persistence the reported RSS doubled and the kernel killed the process.&lt;/p&gt;&#10;&lt;p&gt;This is the Redis fork/copy-on-write memory storm. Redis calls &lt;code&gt;fork()&lt;/code&gt; to spawn a child process for background RDB snapshots, AOF rewrites, and full replication syncs. After the fork, parent and child share pages through copy-on-write. Pages stay read-only until one process writes. If the parent continues serving writes, every modified page is copied. On a write-heavy instance, this can duplicate the entire dataset, pushing RSS to roughly twice the logical data size. Containers with tight memory limits do not see &lt;code&gt;used_memory&lt;/code&gt;; they see RSS. When RSS hits the cgroup ceiling, the OOM killer fires, both processes die, and the instance restarts cold.&lt;/p&gt;</description></item><item><title>Redis latest_fork_usec too high: THP, NUMA, and fork latency</title><link>https://www.netdata.cloud/guides/redis/redis-latest-fork-usec-high/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/redis/redis-latest-fork-usec-high/</guid><description>&lt;p&gt;&lt;code&gt;INFO stats&lt;/code&gt; shows &lt;code&gt;latest_fork_usec&lt;/code&gt; in the hundreds of milliseconds. Every &lt;code&gt;fork()&lt;/code&gt; blocks the single event loop, so during that window no commands are processed. Clients time out, replicas disconnect, and a full resync can trigger another fork, creating a loop of latency and reconnection storms. A normal fork costs roughly 10-20ms per gigabyte of resident memory with Transparent Huge Pages disabled. If you are seeing 10-100x that, the culprit is usually THP, NUMA, or memory overcommit policy.&lt;/p&gt;</description></item><item><title>Redis maxmemory not set: why every production instance needs a memory limit</title><link>https://www.netdata.cloud/guides/redis/redis-maxmemory-not-set/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/redis/redis-maxmemory-not-set/</guid><description>&lt;p&gt;A Redis instance with &lt;code&gt;maxmemory&lt;/code&gt; set to 0 has no memory limit. On 64-bit builds, this is the default. Redis will keep allocating until the OS runs out of physical RAM, at which point the Linux OOM killer terminates the process. No warning, no graceful degradation, no eviction. The process simply vanishes.&lt;/p&gt;&#10;&lt;p&gt;What follows is a predictable restart cycle. Redis starts up, loads the last RDB snapshot (which may be stale or large enough to take minutes), and rejects all data commands during the loading phase. Applications see a cold cache with a 100% miss rate. They hammer the backing database to repopulate Redis. Clients reconnect in a thundering herd. If the dataset that caused the OOM is still being written, the instance hits the wall again and the cycle repeats.&lt;/p&gt;</description></item><item><title>Redis memory pressure spiral: eviction thrashing and how to break it</title><link>https://www.netdata.cloud/guides/redis/redis-memory-pressure-spiral/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/redis/redis-memory-pressure-spiral/</guid><description>&lt;p&gt;Redis latency climbs, CPU saturates, and cache hit rate falls. &lt;code&gt;evicted_keys&lt;/code&gt; rises while application writes increase. The backend database gets hammered. This is not a simple capacity shortage; it is a memory pressure spiral. Redis has reached &lt;code&gt;maxmemory&lt;/code&gt; and started evicting keys. The application responds to cache misses by re-fetching from the origin and writing back to Redis. Those writes trigger more evictions, which cause more misses, which cause more writes. Redis does maximum work for minimum value.&lt;/p&gt;</description></item><item><title>Redis monitoring checklist: the signals every production instance needs</title><link>https://www.netdata.cloud/guides/redis/redis-monitoring-checklist/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/redis/redis-monitoring-checklist/</guid><description>&lt;p&gt;Redis can return PONG while replicating hours behind, during an OOM kill in a background save, or while a KEYS command wedges the event loop. This checklist structures monitoring into four maturity levels. Level 1 is the survival floor. Level 2 adds workload and resource awareness. Level 3 introduces leading indicators that catch degradation before it becomes an incident. Level 4 exposes allocator and encoding internals for granular diagnostics.&lt;/p&gt;&#10;&lt;p&gt;Work through the levels in order. Most production incidents are preventable with Level 2 signals that teams never configure. All metrics below are available via standard Redis commands.&lt;/p&gt;</description></item><item><title>Redis OOM command not allowed when used memory &gt; 'maxmemory' - causes and fixes</title><link>https://www.netdata.cloud/guides/redis/redis-oom-command-not-allowed/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/redis/redis-oom-command-not-allowed/</guid><description>&lt;p&gt;Redis returns &lt;code&gt;(error) OOM command not allowed when used memory &amp;gt; 'maxmemory'&lt;/code&gt;. The server stays online; reads succeed, writes fail. If the client library suppresses errors, the first symptom may be missing data or backend load spikes. This occurs when &lt;code&gt;used_memory&lt;/code&gt; reaches &lt;code&gt;maxmemory&lt;/code&gt; and the eviction policy cannot free space. Under &lt;code&gt;noeviction&lt;/code&gt;, Redis rejects every write and keeps all keys. Under &lt;code&gt;volatile-*&lt;/code&gt;, the same happens when no keys carry a TTL. Monitoring often misses this because &lt;code&gt;evicted_keys&lt;/code&gt; stays at zero while &lt;code&gt;used_memory&lt;/code&gt; sits just below the limit.&lt;/p&gt;</description></item><item><title>Redis OOM-killed by the kernel: RSS, overcommit, and recovery</title><link>https://www.netdata.cloud/guides/redis/redis-out-of-memory-oom-killed/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/redis/redis-out-of-memory-oom-killed/</guid><description>&lt;p&gt;Redis reports &lt;code&gt;used_memory&lt;/code&gt; at 60% of &lt;code&gt;maxmemory&lt;/code&gt;, then disappears. The container status is &lt;code&gt;OOMKilled&lt;/code&gt;, or &lt;code&gt;dmesg&lt;/code&gt; shows the kernel OOM killer selected &lt;code&gt;redis-server&lt;/code&gt;. The kernel enforces resident memory (RSS), while &lt;code&gt;used_memory&lt;/code&gt; and &lt;code&gt;maxmemory&lt;/code&gt; track logical allocator state. Fragmentation, copy-on-write pages during persistence, and client buffers inflate RSS above the logical figure most operators monitor. When RSS hits the host or cgroup memory ceiling, the kernel terminates the process even though Redis believes it is within limits.&lt;/p&gt;</description></item><item><title>SQL Server Error 701: there is insufficient system memory to run this query</title><link>https://www.netdata.cloud/guides/microsoft-sql-server/microsoft-sql-server-error-701-insufficient-memory/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/microsoft-sql-server/microsoft-sql-server-error-701-insufficient-memory/</guid><description>&lt;p&gt;Error 701 is one of SQL Server&amp;rsquo;s bluntest messages: &amp;ldquo;There is insufficient system memory in resource pool &amp;lsquo;default&amp;rsquo; to run this query.&amp;rdquo; When it fires, the engine could not satisfy an allocation. Queries that were running fine seconds ago start failing, and the failure cascades into application timeouts, retry storms, and a flood of related errors (17890, 8645).&lt;/p&gt;&#10;&lt;p&gt;The error itself tells you very little. It does not say whether the buffer pool is starved, whether a single query is hoarding a memory grant, whether an Extended Events ring buffer has eaten 50 GB, or whether the OS is reclaiming memory from a VM balloon driver. Investigate by source.&lt;/p&gt;</description></item><item><title>SQL Server PAGEIOLATCH waits: the buffer pool waiting on slow storage</title><link>https://www.netdata.cloud/guides/microsoft-sql-server/microsoft-sql-server-pageiolatch-waits/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/microsoft-sql-server/microsoft-sql-server-pageiolatch-waits/</guid><description>&lt;p&gt;Your top wait type is &lt;code&gt;PAGEIOLATCH_SH&lt;/code&gt; or &lt;code&gt;PAGEIOLATCH_EX&lt;/code&gt;. Queries that used to be sub-50ms now take seconds. CPU may be low. There are no blocking chains. The buffer pool is waiting on disk.&lt;/p&gt;&#10;&lt;p&gt;PAGEIOLATCH waits are the direct fingerprint of physical I/O. A worker needs an 8KB data page that is not in the buffer pool, takes an in-memory latch on the buffer descriptor, and waits for storage to return the page. When the read completes, the worker continues. Sustained PAGEIOLATCH time means the engine is spending wall-clock time waiting on disk.&lt;/p&gt;</description></item><item><title>SQL Server RESOURCE_SEMAPHORE waits: queries stuck waiting for a memory grant</title><link>https://www.netdata.cloud/guides/microsoft-sql-server/microsoft-sql-server-resource-semaphore-waits/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/microsoft-sql-server/microsoft-sql-server-resource-semaphore-waits/</guid><description>&lt;p&gt;RESOURCE_SEMAPHORE is the wait type SQL Server records when a worker thread cannot get a query memory grant. Before a query runs a sort, hash, or certain joins, the optimizer estimates how much workspace memory it needs and asks the grant pool for it. When the pool is exhausted, parsed-and-optimized queries sit in a queue. To the application they look hung.&lt;/p&gt;&#10;&lt;p&gt;CPU may be low. Disk I/O may be low. Buffer pool may look fine. Users report &amp;ldquo;the database is slow&amp;rdquo; and standard dashboards do not explain why. RESOURCE_SEMAPHORE only shows up clearly if you sample wait stats on a short interval and watch the Memory Grants Pending counter. Both are routinely missed.&lt;/p&gt;</description></item><item><title>vSphere active vs consumed vs granted memory: why the percentage lies</title><link>https://www.netdata.cloud/guides/vmware-vsphere/vmware-vsphere-active-vs-consumed-memory/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/vmware-vsphere/vmware-vsphere-active-vs-consumed-memory/</guid><description>&lt;p&gt;The &amp;ldquo;Memory Usage&amp;rdquo; percentage on a vSphere host summary is one of the most misread signals in infrastructure monitoring. An 85% number that pages you at 3 a.m. may represent a healthy host with no reclamation at all. The same number on a different host may mean VMs are being actively swapped to disk. The percentage alone tells you nothing useful about either state.&lt;/p&gt;&#10;&lt;p&gt;The host summary percentage is computed from consumed memory divided by physical RAM. Operators interpret this as demand, but consumed is not demand. It is a high water mark that includes idle pages. To reason about memory pressure you must read consumed alongside active, granted, and the reclamation signals (balloon, compression, swap). This article explains what each metric actually measures, why consumed greater than active is normal and expected, and how to tell a healthy 85% from a crisis 85%.&lt;/p&gt;</description></item><item><title>vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral</title><link>https://www.netdata.cloud/guides/vmware-vsphere/vmware-vsphere-host-swapping/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/vmware-vsphere/vmware-vsphere-host-swapping/</guid><description>&lt;p&gt;When &lt;code&gt;SWR/s&lt;/code&gt; is sustained above zero on an ESXi host, the VMkernel is actively reading VM memory pages back from &lt;code&gt;.vswp&lt;/code&gt; files on the datastore. That is not a warning state. It is an active performance emergency. Every swapped-in page costs roughly 100x DRAM latency, and the swap I/O itself competes with VM disk I/O on the same datastore, producing a double penalty that degrades every VM on the host simultaneously.&lt;/p&gt;</description></item><item><title>vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM</title><link>https://www.netdata.cloud/guides/vmware-vsphere/vmware-vsphere-memory-ballooning/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/vmware-vsphere/vmware-vsphere-memory-ballooning/</guid><description>&lt;p&gt;You open esxtop, switch to the memory view, and a VM&amp;rsquo;s MCTLSZ column is no longer zero. A few hundred megabytes or several gigabytes, the VMkernel has inflated the vmmemctl balloon driver inside that guest and is forcing the guest OS to hand back memory it thought it owned. From the host&amp;rsquo;s perspective this is gentle reclamation. From the guest&amp;rsquo;s and the application&amp;rsquo;s perspective, it is often the start of a silent performance decline.&lt;/p&gt;</description></item><item><title>ZooKeeper OutOfMemoryError: Java heap space - the OOM that kills the whole ensemble at once</title><link>https://www.netdata.cloud/guides/zookeeper/zookeeper-heap-exhaustion-oom/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://www.netdata.cloud/guides/zookeeper/zookeeper-heap-exhaustion-oom/</guid><description>&lt;p&gt;You grep the ZooKeeper log and find &lt;code&gt;java.lang.OutOfMemoryError: Java heap space&lt;/code&gt;. The process is gone. A minute later another node dies with the same error, then the third. The whole ensemble went down inside a single window, not as a rolling failure. That simultaneity is the signature, not a cascade.&lt;/p&gt;&#10;&lt;p&gt;ZooKeeper holds the entire data tree on the JVM heap: every znode, its data, ACL references, children lists, stat structures, plus session state, watch tables, and request queues. Every ensemble member holds the same tree. Whatever fills the heap on one node fills it on all of them at roughly the same rate, so when the tree finally exceeds the heap they OOM near-simultaneously. This is a single-cause total outage.&lt;/p&gt;</description></item></channel></rss>