Your critical service is lagging. Users are complaining about timeouts. You check your orchestration platform and see the dreaded OOMKilled status on a container. You dive into the node’s logs (dmesg) and confirm it: the kernel’s Out-of-Memory (OOM) killer has claimed another victim. The immediate fix is easy—restart the container, maybe give it more memory—but the real question remains unanswered: why did it happen? Was it a sudden memory leak, a traffic spike, or something more subtle?

With modern Linux distributions running on cgroups v2, the answer is often more nuanced than a simple memory limit breach. Cgroups v2 introduced a sophisticated, tiered system for memory management that can cause performance degradation and throttling long before the OOM killer is ever invoked. Understanding this system is the key to moving from reactive firefighting to proactive container memory optimisation.

TL;DR

  • An OOMKilled container is the final stage of memory pressure, but cgroups v2 throttles performance well before that, so the real diagnosis starts earlier than the kill log.
  • cgroups v2 governs memory through three gates: memory.low protects a cgroup from reclaim, memory.high throttles it with allocation stalls, and memory.max triggers the OOM killer.
  • You diagnose issues by reading the cgroup files directly, using memory.events counters for throttle and kill events and PSI in memory.pressure to measure how long tasks stalled waiting for memory.
  • Kubernetes maps these controls onto pods by QoS class, so watching memory.high events and non-zero PSI values lets you catch leaks and degradation before an outage.

From cgroups v1 To v2: A New Hierarchy For Memory Management

Control groups (cgroups) are a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, etc.) of a collection of processes. Container platforms like Docker and Kubernetes rely on them to enforce resource limits.

While cgroups v1 had a messy, controller-specific hierarchy, cgroups v2 introduced a single, unified hierarchy that simplifies management. For memory, this change was profound. Instead of a single, blunt memory.limit_in_bytes file, v2 introduced a set of controls that create a Memory Quality of Service (QoS) framework. These controls act as three distinct gates that a container’s memory usage must pass through.

The Three Gates Of cgroups v2 Memory Control

Imagine your container’s memory allocation as water flowing into a reservoir. Cgroups v2 sets up three watermarks that trigger different kernel behaviors.

Gate 1: memory.low - The Best-Effort Protection

This is a soft limit. It’s a “best-effort” protection boundary. When a cgroup’s memory usage is below this line, the kernel considers it protected and will try to reclaim memory from unprotected cgroups first.

  • Behavior: No throttling or killing. The kernel will avoid reclaiming memory from this cgroup unless absolutely necessary.
  • Use Case: This is primarily used by orchestrators like Kubernetes to protect pods with a Guaranteed QoS class. For these pods, Kubernetes sets memory.low equal to the memory request, ensuring they are the last to be affected during node-wide memory pressure.

Gate 2: memory.high - The Throttling Gate

This is the most critical control for performance troubleshooting. When a cgroup’s memory usage exceeds memory.high, the kernel starts to aggressively throttle the processes within that cgroup.

  • Behavior: The kernel will try to reclaim memory pages from the cgroup to push its usage back below the memory.high mark. This means that when a process inside the container tries to allocate more memory, its execution is paused while the kernel works to free up space. From the application’s perspective, this manifests as high latency, slowness, or even stalls.
  • Why it Matters: Your application can be performing poorly long before it is ever OOMKilled. If your service is slow but not crashing, it’s very likely hitting the memory.high throttle. This is the “slowness before death” phase that is often invisible without the right monitoring.

Gate 3: memory.max - The Hard Limit & The OOM Killer

This is the final, hard cap. It’s the point of no return.

  • Behavior: If a process inside the cgroup attempts to allocate memory that would push the group’s total usage over memory.max, the kernel’s OOM killer is invoked specifically for that cgroup. It will find a process within the cgroup to kill to free up memory. This is what directly results in the OOMKilled status.
  • Use Case: This sets the absolute boundary for your container’s memory usage, preventing a single leaky container from crashing the entire host node.

Your Troubleshooting Toolkit For Memory Issues

To diagnose these issues, you need to inspect the cgroup files directly on the host node.

Step 1: Find Your Container’s Cgroup

First, you need the path to your container’s cgroup directory. You can find this using the container ID. Once you have the container ID from a command like docker ps, you can locate its cgroup path. The path format is usually consistent, often found under /sys/fs/cgroup/, or you can use a tool like systemd-cgls to explore the hierarchy.

Step 2: Read The Signs In The memory.* Files

Once you’re in the cgroup directory, you can read the diagnostic files.

  • memory.current: Shows the current total memory usage. Compare this value to the limits set in memory.high and memory.max to see where you stand.
  • memory.stat: Provides detailed memory statistics. Key fields include:
    • anon: Anonymous memory (not backed by a file, like heap allocations).
    • file: Page cache (memory used to cache file data from disk). High file usage isn’t always bad, but it contributes to the total memory count.
    • slab: Kernel slab allocations on behalf of the cgroup.
  • memory.events: This is a crucial file for diagnostics. It contains counters for key events.
    • high: The number of times the cgroup’s usage has exceeded the memory.high limit. If this number is climbing, your application is being throttled.
    • max: The number of times the cgroup has hit the memory.max limit, triggering an OOM event.
    • oom: The number of processes OOM-killed within the cgroup.

Step 3: Use PSI (Pressure Stall Information) For Proactive Insight

PSI is a modern kernel feature that provides a much clearer view of resource contention. Instead of just showing event counts, it shows the percentage of wall-clock time that processes were stalled waiting for a resource.

For memory, check the memory.pressure file inside your cgroup directory. Inside this file, you will find metrics for some and full pressure, which provide averages over 10, 60, and 300 seconds, along with a running total.

  • some: This value represents the percentage of time where at least one task was stalled waiting for memory. A non-zero some value is a direct, quantitative measure of memory throttling.
  • full: This value represents the percentage of time where all tasks were stalled simultaneously. A non-zero full value indicates severe memory pressure and is a strong predictor of an imminent OOM kill.

The Kubernetes Context: QoS & Pod Eviction

Kubernetes automates the management of these cgroup controls based on the Quality of Service (QoS) class of your pods, which is determined by the requests and limits you set.

  • Guaranteed: requests == limits. The kubelet sets memory.low and memory.max to the same value. These pods are highly protected.
  • Burstable: requests < limits. The kubelet uses memory.low (based on requests) and memory.max (based on limits), with memory.high often set to the limit. These pods can use more memory than they requested but are candidates for throttling or killing if they exceed their request and the node is under pressure.
  • BestEffort: No requests or limits. These have the lowest priority and are the first to be killed during node-wide pressure.

The kubelet actively monitors these cgroup metrics and PSI. It can make a proactive pod memory eviction decision to kill a pod to preserve node stability, even before the kernel OOM killer is invoked. It uses this data to choose the best candidate for eviction—usually a BestEffort or Burstable pod exceeding its request.

By moving beyond simple OOM kill logs and embracing the rich diagnostic data provided by cgroups v2 and PSI, you can gain a deep understanding of your application’s memory behavior. Monitoring for memory.high events and non-zero PSI some values allows you to detect performance issues and potential memory leaks long before they result in a critical outage.

To make this advanced troubleshooting truly effective, you need a monitoring solution that can collect and visualize these metrics in real-time. Netdata provides per-second visibility into every cgroup v2 control file and PSI metric, automatically, turning complex kernel data into actionable dashboards. Get started with Netdata for free to stop guessing and start diagnosing your container memory issues with precision.

Diagnosing Linux cgroups FAQs

What Does OOMKilled Mean For A Container?

OOMKilled means the Linux kernel’s Out-of-Memory killer terminated a process inside your container to free memory. Under cgroups v2 it happens at a specific gate: when a process tries to allocate memory that would push the cgroup’s total usage past its memory.max hard limit, the kernel invokes the OOM killer scoped to that cgroup and kills a process within it. Your orchestrator then surfaces the OOMKilled status. It’s the final stage of memory pressure, and by the time you see it the container has usually been struggling for a while.

What Is The Difference Between cgroups v1 And v2 For Memory?

cgroups v1 used a messy, controller-specific hierarchy with a single blunt control, memory.limit_in_bytes. cgroups v2 replaced that with a unified hierarchy and a tiered Memory Quality of Service framework. Instead of one hard limit, v2 gives you three distinct gates, memory.low, memory.high, and memory.max, that trigger different kernel behaviors as usage climbs. That shift matters because v2 can throttle a container’s performance long before any hard limit is breached, which v1’s single limit couldn’t express.

What Are The Three Memory Gates In cgroups v2?

cgroups v2 sets three watermarks on a container’s memory. memory.low is a soft, best-effort protection boundary: below it, the kernel avoids reclaiming the cgroup’s memory unless absolutely necessary. memory.high is the throttling gate: cross it and the kernel aggressively reclaims pages and pauses allocations, which the app feels as latency or stalls. memory.max is the hard cap: exceed it and the cgroup’s OOM killer fires. Together they form a tiered system where performance degrades gradually before any container is killed.

What Is The Difference Between memory.high And memory.max?

They sit at different points on the memory curve. memory.high is a throttle: when usage crosses it, the kernel reclaims pages and pauses the container’s allocations to push usage back down, so the application slows but keeps running. memory.max is the hard ceiling: cross it and the kernel’s OOM killer terminates a process, producing the OOMKilled status. The practical takeaway is that memory.high causes the “slowness before death” phase, while memory.max causes the death itself.

How Do I Check Why A Container Was OOMKilled?

Inspect the cgroup files directly on the host node. First find the container’s cgroup directory under /sys/fs/cgroup/ using its container ID from docker ps, or explore the hierarchy with systemd-cgls. Then read the diagnostic files: memory.current for current usage against the limits, memory.stat for a breakdown of anonymous, page-cache, and slab memory, and memory.events for the counters that matter. In memory.events, a climbing high count means throttling, while max and oom counts confirm hard-limit hits and kills.

What Is PSI And How Does It Help Diagnose Memory Pressure?

PSI (Pressure Stall Information) is a kernel feature that reports the percentage of wall-clock time processes spent stalled waiting for a resource, rather than just raw event counts. For memory, read the memory.pressure file in the cgroup directory, which gives some and full metrics averaged over 10, 60, and 300 seconds. A non-zero some value means at least one task was stalled waiting for memory, a direct measure of throttling. A non-zero full value means all tasks stalled at once and is a strong predictor of an imminent OOM kill.

How Does Kubernetes Set cgroup Memory Limits By QoS Class?

The kubelet maps a pod’s requests and limits onto cgroup controls based on its QoS class. Guaranteed pods (requests equal to limits) get memory.low and memory.max set to the same value, making them highly protected. Burstable pods (requests below limits) get memory.low from the request and memory.max from the limit, with memory.high often set to the limit, so they can burst but risk throttling under pressure. BestEffort pods have no requests or limits and the lowest priority, so they’re killed first.

How Can I Detect Container Memory Problems Before An OOM Kill?

Stop waiting for the OOMKilled log and watch the earlier signals instead. Track the high counter in memory.events: if it’s climbing, your container is already being throttled at the memory.high gate. Watch PSI in memory.pressure, where a non-zero some value quantifies throttling and a rising full value warns of an imminent kill. Compare memory.current against the limits to see how much headroom is left. Monitoring these continuously catches performance degradation and potential leaks long before they become a crash.