ZFS ARC shrinking below c_max: reading memory pressure before latency hits

You have a ZFS box that “got slow” and nobody can say why. The disks are fine, the pool is ONLINE, no scrub is running, and zpool iostat shows nothing obviously wrong. Then you look at /proc/spl/kstat/zfs/arcstats and the ARC is sitting at a fraction of its configured maximum, the target size c keeps ratcheting downward, and arc_no_grow is 1.

This is not an ARC problem. The kernel is reclaiming ARC memory because something else wants it. It is the leading edge of the memory-pressure cascade: a smaller ARC means more reads go to disk, more in-flight I/O means more kernel buffers, more buffers mean more memory pressure, and the ARC shrinks further. By the time application read latency spikes or the OOM killer fires, you are already deep in the feedback loop.

The failure mode is visible in four numbers, hours before it hurts. This article covers how to read those numbers, how to confirm what is actually consuming the memory, and how to break the loop.

What this means

The ARC is ZFS’s primary read cache, and it lives in kernel memory outside the Linux page cache. It grows and shrinks dynamically in response to memory pressure. The four size values in arcstats tell you where it stands:

• size: current ARC size in bytes.
• c: current target size. ZFS adjusts this continuously between the floor and the ceiling based on cache behavior and memory pressure.
• c_max: the hard ceiling. Set by zfs_arc_max, or by the platform default if you never configured it.
• c_min: the floor. The ARC will not shrink below this.

In healthy operation, size tracks close to c, and c sits at or near c_max. The signal is the divergence: size well below c_max, c oscillating downward over time, and arc_no_grow set. That combination means the kernel’s memory reclaim subsystem is actively draining the ARC to satisfy other allocations. ZFS is cooperating. The question is why the system needs the memory, and whether the drain stops before the cascade starts.

flowchart TD
  A[Application memory demand grows] --> B[Kernel reclaims ARC pages]
  B --> C[ARC size falls below c_max]
  C --> D[arc_no_grow set, target c ratchets down]
  D --> E[Cache hit ratio drops]
  E --> F[More reads served from disk]
  F --> G[More in-flight I/O and kernel buffers]
  G --> H[More memory pressure]
  H --> B
  C -. leading indicator, visible here first .-> I[Watch size vs c vs c_max]
  H --> J[Swap usage rises or OOM killer fires]

Two things make this signal easy to misread. First, on Linux the ARC appears as “used” memory in free output, not as cache, even though it is reclaimable. Operators see low “free” memory and either panic incorrectly or ignore it incorrectly. Second, the ARC does not shrink instantly when memory is demanded. Reclaim has latency, so under sudden pressure the system can hit swap or OOM before the ARC has finished releasing pages. For the opposite failure mode, where the ARC consumes everything because no cap was ever set, see ZFS ARC using all memory: the Linux default that eats your RAM.

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# The four ARC size numbers, plus the no-grow flag
grep -E "^(size|c|c_max|c_min|arc_no_grow)\s" /proc/spl/kstat/zfs/arcstats

# Is the ARC being throttled for memory? Non-zero and rising = active pressure
grep "^memory_throttle_count" /proc/spl/kstat/zfs/arcstats

# Human-readable summary if the tools are installed
arcstat 1 5
arc_summary

# System memory view: what the kernel thinks is available
grep -E "MemAvailable|MemFree|SwapTotal|SwapFree" /proc/meminfo

# Who is actually holding memory
ps aux --sort=-%mem | head -15

# Has the OOM killer already fired? (may need sudo for dmesg)
dmesg -T | grep -i -E "out of memory|oom-kill|killed process"

# What is the configured ARC ceiling? 0 means the platform default
cat /sys/module/zfs/parameters/zfs_arc_max

The two most important single reads: the ratio of size to c_max, and whether arc_no_grow is 1. An ARC at 95% of c_max with arc_no_grow clear is healthy. An ARC at 30% of c_max with arc_no_grow set is a system under active memory reclaim, and it will get worse until the demand stops.

How to diagnose it

1. Confirm the divergence. Pull size, c, c_max, and c_min from arcstats. If size is near c_max and c is stable, this article does not apply to your problem. If size is well below c_max and c is falling, you have active reclaim.

2. Check the direction of c over time. A single snapshot is ambiguous because c also moves for cache-behavior reasons (ghost list adaptation). What confirms memory pressure is c oscillating or ratcheting downward across minutes to hours while c_max is unchanged. Sample it a few times:

   # Watch the target size move (watch ships with procps; or loop grep in a while/sleep)
   watch -n 10 'grep -E "^(size|c)\s" /proc/spl/kstat/zfs/arcstats'
   

3. Check arc_no_grow. When this is 1, the ARC has been told by the memory management subsystem not to grow even if cache behavior says it should. This pins the ARC and explains a low hit ratio that never recovers.

4. Check memory_throttle_count. If this counter increments between samples, ZFS is actively throttling I/O because of memory pressure. The cascade is no longer theoretical.

5. Correlate with system memory. Read MemAvailable and swap usage from /proc/meminfo. ARC shrink plus declining MemAvailable plus rising swap is the full cascade signature. ARC shrink with plenty of MemAvailable points elsewhere: for example, a recently lowered zfs_arc_max is intentional shrink, not pressure.

6. Find the consumer. ps aux --sort=-%mem gives the current snapshot, but for a slow squeeze you need history: which process’s RSS grew over the same window in which c declined? If you have no historical per-process data, that gap is your first monitoring fix.

7. Check for OOM evidence. dmesg -T | grep -i oom. If the kernel has already killed processes, the ARC did not shrink fast enough to absorb the pressure, and future bursts will do the same until you add headroom.

8. Check the read path consequences. A declining ARC hit ratio (hits vs misses in arcstats) combined with rising disk read IOPS confirms the cascade has reached I/O, not just memory accounting.

Metrics and signals to monitor

The correlation that matters: ARC size falling, c falling, MemAvailable falling, disk read IOPS rising, all in the same time window. Each signal alone is ambiguous. Together they are unambiguous.

Fixes

Reduce or contain the memory consumer

If a specific process is the cause, fix it at the source: restart the leaking service, cap the database’s buffer pool, or tighten container memory limits so collective overcommit cannot recur. This is the only fix that addresses the root cause. Everything else is headroom management.

Set an explicit zfs_arc_max

If zfs_arc_max is 0 (platform default) or set too high for a shared machine, bound the ARC so applications get guaranteed room. The change takes effect immediately:

# Cap the ARC at, for example, 16 GiB (adjust to your machine)
echo 17179869184 > /sys/module/zfs/parameters/zfs_arc_max

The ARC will shrink to the new limit. The tradeoff is real: a smaller ceiling means a smaller cache and a lower hit ratio for working sets larger than the cap. On a dedicated storage box, a generous cap is correct. On a host shared with databases or containers, the ARC must be explicitly bounded or every new deployment becomes an ARC incident. Persist the setting in /etc/modprobe.d/zfs.conf so it survives reboot:

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=17179869184

Sizing guidance from operational practice: leave at least 20-25% of physical RAM for the OS and applications, more on hosts running memory-hungry services alongside ZFS.

Add headroom for reclaim latency

The ARC cannot release memory instantly; it must walk and free internal structures. Under a sudden allocation burst, the kernel can exhaust RAM and invoke the OOM killer before the ARC finishes shrinking. If this host has already OOM-killed processes, lowering zfs_arc_max to create standing headroom is more reliable than relying on reactive reclaim. Swap can absorb short bursts, but on a ZFS host treat swap as a buffer for reclaim latency, not as a substitute for RAM.

What not to do

Do not drop caches to “help” the ARC. Do not raise zfs_arc_min to force the ARC to stay large: that converts your memory pressure into guaranteed application OOMs, because ZFS will refuse to give memory back. Do not restart ZFS or remount datasets; the ARC shrink is a symptom, and the pressure will resume within hours.

Prevention

• Set zfs_arc_max on every host that runs anything besides ZFS. This is the single highest-value prevention step. An unbounded ARC on a shared machine is a latent incident.
• Trend the four numbers. size, c, c_max, c_min, plus arc_no_grow and memory_throttle_count, exported continuously to a time-series system. Point-in-time arcstats reads during an incident tell you where you are; trends tell you when it started and how fast it is moving.
• Alert on the divergence, not the absolute value. size < 50% of c_max sustained, or c declining over a multi-hour window, catches the pressure phase before the latency phase.
• Track per-process RSS history. Diagnosing a slow ARC squeeze without process memory history is guesswork.
• Treat OOM kills on a ZFS host as ARC misconfiguration until proven otherwise. The kernel can kill applications while gigabytes of reclaimable ARC exist, because reclaim is not instant and the OOM killer’s view of ARC is imperfect.

How Netdata helps

• Netdata collects the full arcstats kstat, so size, c, c_max, c_min, arc_no_grow, and memory_throttle_count are graphed on one dashboard at per-second resolution, making the downward ratchet of c visible as it happens rather than after.
• ARC size and hit ratio are correlated automatically with system-level MemAvailable and swap usage, so the memory-pressure cascade signature (ARC down, available down, swap up, disk reads up) appears as one story instead of four separate charts.
• Per-process and per-cgroup memory trends let you answer “which process grew while the ARC shrank” without having instrumented it in advance.
• Anomaly detection on the ARC size and hit-ratio series flags a sustained divergence from baseline, which is the alert shape this failure mode actually needs, since absolute thresholds false-fire on legitimate cache adaptation.
• Disk read IOPS on pool devices alongside ARC hit ratio makes the cache-starvation consequence explicit: hit ratio falling while read IOPS climbs is the cascade entering its I/O phase.

Related guides

• ZFS ARC using all memory: the Linux default that eats your RAM
• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• ZFS capacity planning: runway estimation before the pool fills
• ZFS pool ONLINE with non-zero errors: why zpool status -x lies
• ZFS capacity cliff: why the pool falls off a performance edge near 80-90% full