ZFS ARC and the OOM killer: applications killed while the cache will not shrink fast enough

Your database or application process just got OOM-killed on a ZFS host that “had plenty of memory.” dmesg shows the OOM killer invoked and a victim chosen, while the ARC was holding tens of gigabytes at the time. The machine was not out of memory. It was out of memory the kernel could get back fast enough.

The ARC is technically reclaimable, but it does not release memory instantly under a sudden allocation spike. The arc_prune shrinker callback has inherent latency. When a large allocation lands faster than the ARC can drain, direct reclaim gives up and the OOM killer fires, and it frequently picks the wrong victim, because the process that triggered the allocation is not necessarily the one the badness score ranks highest.

What this means

On Linux, the ARC lives in kernel slab memory, outside the page cache. free shows it as “used,” not “cached,” even though most of it is evictable cache. Two consequences follow:

1. Reclaim is asynchronous under pressure. When an application allocates memory, the kernel asks the ARC shrinker for pages. If allocation demand outruns the shrinker, the kernel enters the OOM path while the ARC still holds gigabytes.
2. Not all ZFS memory is reclaimable. The DDT (dedup table), metaslab state, and in-flight I/O buffers are slab allocations the kernel cannot evict. On a system with dedup enabled or heavy write load, a slice of “ZFS memory” is permanent until the workload stops.

A second trap: MemAvailable in /proc/meminfo does not count ARC the way it counts page cache, so user-space OOM tools (earlyoom, systemd-oomd) and operators reading free -m see artificially low “available” memory. The host can have 40 GB of evictable ARC and still look like it is about to run out.

flowchart TD
  A[Sudden large allocation
application, container, fork storm] --> B[Kernel direct reclaim]
  B --> C[ARC shrinker invoked
arc_prune starts evicting]
  C --> D{ARC drains fast enough?}
  D -- yes --> E[Allocation succeeds
memory_throttle_count may increment]
  D -- no --> F[OOM killer fires]
  F --> G[Victim killed by badness score
often not the allocator]
  C -.-> H[Non-reclaimable ZFS memory stays:
DDT, metaslab state, in-flight IO]

Common causes

Quick checks

All read-only and safe to run during an incident.

# Did the OOM killer actually fire, and what did it kill?
dmesg -T | grep -i -E "out of memory|oom-kill|killed process" | tail -20
journalctl -k --since "1 hour ago" | grep -i oom

# Is ARC capped? 0 means no explicit cap (OpenZFS computes its own default)
cat /sys/module/zfs/parameters/zfs_arc_max
cat /sys/module/zfs/parameters/zfs_arc_min

# Current ARC size, target, and ceiling (bytes)
awk '/^(size|c|c_min|c_max|memory_throttle_count) / {print $1, $3}' /proc/spl/kstat/zfs/arcstats

# Memory picture the kernel actually sees
grep -E "MemTotal|MemFree|MemAvailable|Slab|SReclaimable|SUnreclaim" /proc/meminfo

# Who is holding memory right now
ps aux --sort=-%mem | head -10

# Rule out cold-start noise: ARC is legitimately large after boot
uptime

Reading the output: if zfs_arc_max is 0 (or close to total RAM), size is near MemTotal, MemAvailable was near zero, and dmesg shows a kill, you have the pattern. memory_throttle_count incrementing is the leading indicator: the ARC is being throttled by memory pressure before the OOM happens.

On a freshly booted host the ARC grows aggressively while other allocations settle, so treat the pattern as suspect on any host with uptime under about 10 minutes.

How to diagnose it

1. Confirm the kill and the victim. Pull the OOM report from dmesg or journalctl -k. Note which process was killed and which process triggered the allocation. If they differ, that fits this pattern: the killer scores by memory footprint, not by fault.
2. Check whether ARC had headroom it failed to release. Compare ARC size in arcstats (or your metrics history) against MemTotal at the time of the kill. ARC at 70-90% of RAM with no cap set is the smoking gun.
3. Check the throttle counter. If memory_throttle_count was incrementing in the minutes before the OOM, memory pressure was real and sustained, not a one-off spike.
4. Identify the allocator. ps history, container runtime events, deploy timelines. A new service, a raised heap limit, or a batch job landing on the host is the usual trigger.
5. Rule out non-reclaimable ZFS memory. If dedup is enabled (zpool get dedup <pool>), the DDT lives in ARC and cannot be evicted like ordinary cache. Heavy in-flight write I/O also pins slab. If slab stays high after the ARC target drops, capping the ARC alone will not fully fix it.
6. Check for user-space OOM tooling. If earlyoom or systemd-oomd made the kill, check whether it acted on MemAvailable, which undercounts reclaimable ARC. The kill may have been premature even though the host was fine.

Metrics and signals to monitor

Fixes

Emergency: stop the bleeding now

Set an explicit ARC cap at runtime. It takes effect immediately, but understand what “takes effect” means: lowering zfs_arc_max below the current ARC size does not force an instant shrink. The ARC contracts as memory or eviction pressure arrives. The pressure from the workload itself usually does it, but do not treat the sysfs write as synchronous.

# Example: cap ARC at 16 GiB on a 32 GiB host
echo $((16*1024*1024*1024)) > /sys/module/zfs/parameters/zfs_arc_max

If a user-space OOM tool made the kill, reconfigure or suspend it until it accounts for ARC correctly, otherwise it will keep killing processes while gigabytes of cache sit idle.

Cap the ARC permanently

Set zfs_arc_max in /etc/modprobe.d/zfs.conf so it survives reboots:

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=17179869184

Sizing guidance: leave 20-25% of RAM for the OS and applications, so on most Linux hosts the ARC cap should not exceed roughly 75-80% of RAM. On a shared host (database plus ZFS), budget the application’s working set first and cap the ARC to what remains. On a dedicated storage host a larger share is fine, but set the cap explicitly either way.

Also check zfs_arc_min. If a previous tuning session raised it, the ARC cannot shrink below that floor and the OOM risk returns no matter what zfs_arc_max says.

Deal with non-reclaimable memory

If dedup is enabled, the DDT (roughly 320 bytes per deduplicated block) must live in memory and is not evictable like cache. If DDT pressure is part of the problem, the durable fix is migrating data off dedup datasets, not tuning ARC. Heavy in-flight I/O pinning slab is usually a write pipeline problem; check TXG sync times and dirty data before blaming the ARC.

Reduce the allocation spike

Cap the application side too: container memory limits, database shared buffers, JVM heap. The failure requires both an unbounded ARC and a fast allocator; fixing either side breaks the pattern.

Prevention

• Always set zfs_arc_max explicitly on Linux. The default lets the ARC consume most of RAM. Recent OpenZFS releases raised the default ARC ceiling well above the historical 50% of RAM , so systems upgraded from older versions may newly be at risk even if they were stable for years.
• Alert on memory_throttle_count increments before any OOM happens. It is the earliest reliable signal in this failure pattern.
• Trend ARC size, c, and MemAvailable together. The dangerous configuration is ARC near RAM with no cap; you can see it weeks before it bites.
• Budget memory on shared hosts deliberately. Application working set + OS + ARC cap + buffer must be less than physical RAM.
• Review zfs_arc_min whenever you review zfs_arc_max.
• Validate user-space OOM tooling against ZFS. If earlyoom or systemd-oomd watches MemAvailable, test its behavior under ARC load or it will produce false kills.

How Netdata helps

• Netdata collects /proc/spl/kstat/zfs/arcstats per second, so ARC size, target c, c_max, and hit ratio are trended continuously rather than sampled after the fact.
• memory_throttle_count is charted as a counter, so you can alert on the first increment instead of the first OOM.
• System memory charts (MemAvailable, slab, reclaimable vs unreclaimable) sit next to the ARC charts, so you can see the ARC failing to drain at the same moment allocation demand spikes.
• Kernel log monitoring surfaces OOM kills as events you can correlate against the ARC and memory timeline in the same dashboard.
• Per-second resolution lets you distinguish “ARC shrank but too late” from “ARC never shrank,” which determines whether the fix is a lower cap or an application-side limit.

Related guides

• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• ZFS capacity cliff: why the pool falls off a performance edge near 80-90% full
• ZFS No space left on device: ENOSPC, the slop reserve, and the pool you cannot delete from