ZFS ARC using all memory: the Linux default that eats your RAM

You run free -h on a ZFS host and see 58 of 64 GiB “used”, with almost nothing in buffers/cache. Your application processes account for maybe 8 GiB. Something is eating the machine, and you start hunting for a leak.

There is no leak. The missing memory is the ZFS ARC (Adaptive Replacement Cache), ZFS’s primary read cache in kernel memory. On Linux the ARC lives outside the kernel page cache, so free and top report it as used slab, not as reclaimable cache. Operators who do not know this tune swappiness, add swap, or kill innocent processes to “free” memory that was never in danger.

This symptom has two distinct halves. The first is a reporting problem: the ARC looks like a memory shortage when it is actually the cache working as designed. The second is a real risk: with no explicit cap, the ARC can grow large enough that applications genuinely starve and the OOM killer fires, because the ARC is reclaimable but not instantly reclaimable. This guide covers how to tell the two apart and how to set zfs_arc_max correctly.

What this means

The ARC caches both data and metadata blocks, and grows and shrinks in response to memory pressure. Left alone, it consumes available memory aggressively, because from ZFS’s perspective unused RAM is wasted cache.

The default ceiling depends on your OpenZFS version. Before OpenZFS 2.3.0, Linux defaulted the maximum ARC size to 50% of system RAM. Since OpenZFS 2.3.0, every platform uses the larger of all_system_memory - 1 GiB and 5/8 x all_system_memory, which approaches nearly all of RAM on larger hosts. Either way, if you never set zfs_arc_max, the effective ceiling is far above what a shared application host can tolerate.

The reclaim problem is the dangerous part. The ARC shrinks under kernel memory pressure, but with latency. A sudden large allocation can trigger the OOM killer before the ARC has had time to shrink. Compounding this, the ARC is not accounted as available memory the way page cache is: it shows up in slab usage, and a long-standing OpenZFS issue (#10255) tracks the fact that ARC is not reflected in MemAvailable. Tools like earlyoom and systemd-oomd therefore see falsely low available memory on ZFS hosts.

flowchart TD
  A["free/top shows almost no free memory"] --> B{"ARC size large in arcstats?"}
  B -- "no" --> D["Real application or kernel memory use - not ARC"]
  B -- "yes" --> C{"memory_throttle_count rising, or OOM kills in dmesg?"}
  C -- "no" --> E["Healthy ARC caching - by design, no action"]
  C -- "yes" --> F["Genuine pressure - cap the ARC"]
  F --> G["Runtime: sysfs parameter, Persistent: /etc/modprobe.d/zfs.conf"]

Common causes

Quick checks

All of these are read-only and safe on a production host.

# What is the ARC allowed to grow to? 0 means "no explicit cap"
cat /sys/module/zfs/parameters/zfs_arc_max

# Current ARC size, target, and hard limits (bytes)
grep -E "^(size|c|c_min|c_max) " /proc/spl/kstat/zfs/arcstats

# Is ZFS actively throttling I/O due to memory pressure?
grep "^memory_throttle_count" /proc/spl/kstat/zfs/arcstats

# How much of what free calls "used" is actually slab (where the ARC lives)?
grep -E "^(MemTotal|MemAvailable|Slab|SReclaimable)" /proc/meminfo

# Has the OOM killer actually fired?
dmesg | grep -i -E "out of memory|oom-kill"

# Human-readable summary of ARC state
arc_summary

Interpretation: if size is near c_max, memory_throttle_count is flat, and there are no OOM events, you are looking at healthy caching. If memory_throttle_count is incrementing or dmesg shows OOM kills, the ARC (or something else) is genuinely starving applications.

How to diagnose it

1. Establish uptime context. If the machine booted recently, a growing ARC is just the cache warming. Only diagnose pressure on hosts up long enough to reach steady state (the playbook uses uptime > 600 seconds as the noise floor).
2. Confirm the memory is ARC. Compare size from arcstats against the gap between your application RSS total and physical RAM. If ARC size explains the “missing” memory, stop hunting for a leak.
3. Check for real pressure. A single snapshot of memory_throttle_count is meaningless; watch it over a minute or two. Incrementing means ZFS is actively throttling I/O because of memory pressure, which is real, not cosmetic.
4. Check for OOM evidence. Look in dmesg and journalctl -k. If the OOM killer fired and the victim was an application while ARC size was large, the ARC did not shrink fast enough. That is the failure mode a cap prevents.
5. Decide which situation you are in. Large ARC, flat throttle counter, no OOM: healthy, optionally cap for headroom. Large ARC plus throttle or OOM: cap now. Shrinking ARC plus rising disk reads: some other process is the memory consumer; find it before blaming ZFS.

Metrics and signals to monitor

Fixes

Cap the ARC at runtime

# Example: cap the ARC at 32 GiB (takes effect immediately)
echo 34359738368 > /sys/module/zfs/parameters/zfs_arc_max

The new ceiling applies immediately, with two caveats. First, the ARC is reclaimable but not instantly: if the current size is above your new cap, it shrinks as memory pressure and eviction push it down, not all at once. Second, the parameter may not accept being set back to 0 (uncapped) while the module is loaded; returning to the default can require a reboot or module reload.

Sizing guidance from the operational playbook: the ARC should leave 20-25% of system RAM for the OS and applications, and should not exceed 80% of RAM. On a dedicated storage host, 75% of RAM is a common target. On a shared host running databases or application servers, be more conservative and size from the application’s working set upward.

Make the cap persistent

The sysfs write does not survive a reboot. Set it in module configuration:

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=34359738368

On root-on-ZFS systems the module is loaded from the initramfs, so regenerate it (for example update-initramfs -u on Debian/Ubuntu, dracut on RHEL-family) or the cap will not apply early in boot.

One constraint worth knowing: zfs_arc_max must be above the ARC minimum (c_min; check grep ^c_min /proc/spl/kstat/zfs/arcstats on your host). Values below that floor are rejected, and on recent OpenZFS releases the module logs a warning rather than silently ignoring the setting.

When the ARC is not the problem

If size is well below c_max and memory is still tight, the ARC is the victim, not the cause. Find the real consumer (ps aux --sort=-%mem), and if it is a legitimate workload, size zfs_arc_max explicitly so the split between ARC and application is deliberate rather than fought over by the kernel at 3 a.m. Setting the cap too low is a real tradeoff: hit ratio falls and read latency rises. Treat the cap as a budget decision, not a reflex.

One sharp edge: do not put swap on a ZFS zvol or dataset. Under memory pressure, ZFS itself needs memory to service swap I/O, which can deadlock. Use a separate non-ZFS partition for swap.

Prevention

• Set zfs_arc_max on every ZFS host at provisioning time. The default ceiling (nearly all of RAM on OpenZFS 2.3.0+ on larger hosts) is only safe on dedicated storage appliances, and even there it deserves an explicit decision.
• Alert on memory_throttle_count increments, not on ARC size. A large ARC is normal. Throttling is the signal that pressure is real.
• Monitor the ARC hit ratio alongside the cap. If you lower the cap and the hit ratio falls off a cliff, you traded an OOM risk for a latency problem; revisit the split.
• Be skeptical of free on ZFS hosts. Train the team to read arcstats before declaring a memory emergency. This one habit prevents most of the misdiagnosis this symptom causes.

How Netdata helps

• ARC size vs target vs ceiling in one view. Netdata charts size, c, and c_max from arcstats, so “is the ARC at its cap or being squeezed?” is a glance, not a grep.
• memory_throttle_count as a rate. Continuous collection separates a flat line (healthy caching) from an incrementing one (real pressure) without manual sampling.
• Hit ratio next to memory pressure. Correlating ARC hit ratio with MemAvailable and slab usage shows whether your RAM is buying cache hits or just sitting large.
• OOM events on the same timeline. Kernel OOM kills overlaid on ARC size and application memory makes the “ARC did not shrink fast enough” failure mode visible instead of inferred from dmesg after the fact.
• Disk reads as the downstream signal. When the ARC is squeezed, read IOPS climb; having all three signals on one dashboard is what distinguishes ARC starvation from a disk problem.

Related guides

• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS No space left on device: ENOSPC, the slop reserve, and the pool you cannot delete from