On Linux, ZFS does not ship with a sane upper bound on the ARC for you. Left alone, the ARC aggressively consumes available memory, and because ARC memory is managed outside the kernel page cache, it shows up as “used” in free even though it is reclaimable. On a shared host, the outcome is familiar: a database or application process gets OOM-killed, and nothing in the storage layer looks wrong.

The fix is one module parameter: zfs_arc_max. The hard part is not setting it. The hard part is choosing a value that protects applications without shrinking the ARC so far that the hit ratio collapses and every read goes to disk. Set it too high and you are back to OOM risk. Set it too low and you have traded a memory problem for a latency problem.

This guide covers how to pick the number, apply it at runtime and persistently, verify it, and tell the difference between “ARC is capped and healthy” and “ARC is capped and starving.”

Why the ARC needs an explicit cap

The ARC grows and shrinks in response to memory pressure, but the shrink path is not instant and does not always win the race. A sudden large allocation from an application can trigger the OOM killer before the ARC has released enough memory. The ARC reclaim path has inherent latency, and non-ARC ZFS memory (in-flight I/O, metadata, the DDT if dedup is enabled) is not reclaimable at all.

Two deployment shapes, two different risks:

  • Dedicated storage host. ZFS is the main tenant. The ARC consuming most of RAM is by design and efficient. A generous cap, commonly around 75% of RAM, leaves room for the OS and avoids pathological edge cases.
  • Shared host. Databases, hypervisors, or application servers run alongside ZFS. Here an uncapped ARC is a standing incident. The ARC must be explicitly bounded so that ARC + application working set + OS never exceeds physical RAM.

If you are on the fully uncapped default and already seeing memory pressure, start with ZFS ARC using all memory and come back here for the sizing work.

How to choose the value

The budget equation:

RAM >= zfs_arc_max + application working set + OS base + 10% buffer

Work it from the right side. Decide what the applications genuinely need under peak load (not their RSS at idle), add the OS baseline, add a 10% buffer for the kernel and transient allocations, and give the ARC what remains.

flowchart LR
  RAM[Physical RAM] --> ARC[zfs_arc_max]
  RAM --> APP[App working set]
  RAM --> OS[OS base]
  RAM --> BUF[10% buffer]
  ARC --> HIT[ARC hit ratio]
  APP --> OOM[OOM risk if squeezed]

Rules of thumb that hold up in production:

  • Dedicated ZFS box: 75% of RAM is a common, defensible cap. Monitor and adjust from there.
  • Shared box: compute the ARC allocation from the budget equation, not from a percentage. The percentage is an output, not an input.
  • Leave 20-25% of RAM for OS and applications at minimum on anything that is not a pure storage appliance.
  • Account for L2ARC headers. If you have L2ARC, its index lives in ARC RAM (roughly 70 bytes per cached block). A large L2ARC silently eats into the cap you thought you had.
  • Dedup changes everything. The DDT consumes about 320 bytes per block and is pinned in ARC. If dedup is on, the DDT’s memory must come out of the ARC budget first.

Do not pick the cap to make free output look comfortable. ARC memory appears as used but is reclaimable; the goal is a guarantee for applications, not a pretty number.

Procedure

1. Establish the current state

# Current cap (0 means no explicit limit)
cat /sys/module/zfs/parameters/zfs_arc_max

# Current ARC size, target, and limits
awk '/^(size|c|c_min|c_max) / {print $1, $3}' /proc/spl/kstat/zfs/arcstats

# Memory picture (remember: ARC shows as used, not available)
grep -E 'MemTotal|MemAvailable' /proc/meminfo

Note the distinction: size is what the ARC currently holds, c is the current dynamic target, and c_max is the hard ceiling that zfs_arc_max controls. Confusing size with c_max is a common misreading.

2. Measure the hit ratio before you change anything

You need a baseline or you will not be able to tell whether the new cap hurt read performance.

# Overall ARC hit ratio
awk '/^hits / {h=$3} /^misses / {m=$3} END {printf "%.2f%%\n", h/(h+m)*100}' /proc/spl/kstat/zfs/arcstats

# Or watch it live
arcstat 1 10

Interpretation is workload-dependent: above 80% is the target for read-heavy workloads, above 60% for mixed, and low hit ratios can be normal for write-heavy or sequential-streaming workloads. A hit ratio near 0% right after boot means nothing; the ARC takes hours to warm.

3. Set the cap at runtime

# Example: cap the ARC at 32 GiB
echo 34359738368 > /sys/module/zfs/parameters/zfs_arc_max

This takes effect immediately and the ARC shrinks to the new limit. It is safe to run on a live system; the ARC will evict cached data to comply, which is why you took the hit-ratio baseline first. Expect read latency to rise temporarily as the cache re-equilibrates below the new ceiling.

4. Persist it across reboots

The sysfs write is lost on reboot. Persist the value in the module configuration:

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=34359738368

The persistent value applies at module load, which in practice means at boot. Always set both: the runtime write fixes the host now, the modprobe file fixes it forever. Skipping the modprobe file is how a well-tuned host regresses silently after the next reboot.

Verifying it works

After applying the cap, confirm all three of these:

  • The ceiling stuck. cat /sys/module/zfs/parameters/zfs_arc_max returns your value, and c_max in arcstats matches it.
  • The ARC is actually using its headroom. size should settle near c_max under normal read load. A size persistently far below c_max means the ARC is being squeezed by memory pressure from somewhere else, which is a different problem.
  • The hit ratio survived. Compare against your baseline after the ARC has had time to rewarm (hours, not minutes). A drop of a few points is the price of a smaller cache. A collapse means you cut too deep.

Watch memory_throttle_count in arcstats as well. If it starts incrementing after your change, I/O is being throttled due to memory pressure and your budget math was too optimistic somewhere.

Common pitfalls

  • Capping too low to be safe. Operators burned by an OOM overcorrect and starve the ARC. The symptom is read latency climbing with no change in the disks. The check is the hit ratio and the size field, not the disks.
  • Capping too high on a shared box. “ARC only grows to what it needs” is wrong on a host with bursty applications. The ARC gets there first and gives memory back slowly. Bound it explicitly.
  • Setting zfs_arc_min too high. A high floor prevents the ARC from shrinking under genuine pressure and can itself cause OOM. Most hosts should leave the floor alone and tune only the ceiling.
  • Double-caching with databases. A database with its own large buffer pool plus a big ARC caches the same blocks twice. For database datasets, consider primarycache=metadata so the ARC caches metadata only and the cap you set actually means something.
  • Judging by free. Low “available” memory with a large ARC is normal, not a leak. Judge by hit ratio, read latency, and whether applications are being OOM-killed.
  • Changing the cap and walking away. Without the baseline comparison, you cannot know whether you traded an OOM risk for a read-performance regression. ARC sizing defaults also vary by OpenZFS version, so re-validate after upgrades.

Signals to monitor

SignalWhy it mattersWarning sign
ARC size vs c_max (arcstats)Shows whether the ARC can use its allowancesize persistently far below c_max means external memory pressure
ARC hit ratio (hits/misses)Direct measure of whether the cap starved readsSustained drop below baseline after a cap change; below 80% on read-heavy workloads
memory_throttle_countCounts I/O throttled due to memory pressureAny sustained incrementing
MemAvailable (/proc/meminfo)The real headroom applications haveTrending toward zero while ARC sits at its cap
OOM kills in dmesg/journalctlProof the cap (or lack of one) failedAny OOM event involving an application process
Read latency (zpool iostat -l)Downstream cost of a too-small ARCRising read latency alongside a falling hit ratio

How Netdata helps

  • Netdata collects the ARC kstats (size, c, c_max, hits, misses, memory_throttle_count) per second, so a cap change shows up immediately as a step in size and you can watch the hit ratio respond in the same view.
  • Correlating ARC hit ratio with pool read latency answers “did the cap hurt reads?” in one screen instead of two SSH sessions.
  • Plotting ARC size next to system MemAvailable and application RSS makes the memory budget visible, which is the difference between guessing at zfs_arc_max and deriving it.
  • OOM events and a rising memory_throttle_count are alertable conditions, so a mis-sized cap pages you before users notice the latency.
  • Long retention on the hit ratio gives you the pre-change baseline this whole procedure depends on, even if you forgot to measure it manually.