ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM

Your monitoring says the ZFS ARC hit ratio dropped, or reads got slow and someone traced it to cache misses. The number itself tells you almost nothing until you know three things: which hit ratio you are looking at (demand vs prefetch), what the workload’s baseline is, and whether the ARC is actually at its intended size.

A low ARC hit ratio is one of the most misread signals in ZFS operations. Near-zero hit ratio after a reboot is normal. A 30% demand hit ratio on a database doing random reads over a dataset larger than RAM can be perfectly healthy. A file server that has run at 96% for months dropping to 78% is an incident. The absolute value is meaningless; the deviation from baseline is the signal.

What this means

The ARC (Adaptive Replacement Cache) is ZFS’s primary read cache in kernel memory. Reads that hit the ARC are served from RAM in microseconds; misses go to disk (or L2ARC, if present) in milliseconds. The hit ratio is the fraction of reads served from cache.

The naive calculation from /proc/spl/kstat/zfs/arcstats is hits / (hits + misses). That mixes two very different populations:

• Demand hits and misses: the application asked for this block. Demand misses are real read latency.
• Prefetch hits and misses: the readahead engine guessed and fetched ahead. Prefetch misses mean readahead guessed wrong; they do not directly slow the application.

An aggregate ratio that includes prefetch can hide a demand-read problem, and a low aggregate can look alarming when the demand ratio is fine. You want the demand hit ratio, split into demand data vs demand metadata, because those two splits point at different root causes.

There are three benign reasons for a low hit ratio and one that indicates a real problem:

1. Cold cache: after boot, pool import, or an ARC flush, every first read misses. The ratio climbs over hours as the working set loads.
2. Expected cache-hostile workload: sequential streaming, scrubs, resilvers, and random reads over a dataset much larger than RAM will never have a high hit ratio.
3. Working set growth: hot data no longer fits, so previously cached blocks are evicted before reuse.
4. ARC shrinkage under memory pressure: the ARC was forced to return memory to the kernel. This is the one that escalates, because it creates a feedback loop: smaller ARC, more disk I/O, more in-flight I/O buffers, more memory pressure.

flowchart TD
  A[Low ARC hit ratio alert] --> B{Uptime or import recent?}
  B -- yes --> C[Cold cache - wait and trend]
  B -- no --> D{Scrub or resilver running?}
  D -- yes --> E[Expected - hit ratio recovers after]
  D -- no --> F{Workload changed?}
  F -- sequential or random-over-big-dataset --> G[Expected for this access pattern]
  F -- no change --> H{ARC size near c_max?}
  H -- no, size shrank --> I[Memory pressure: find the consumer]
  H -- yes --> J[Working set exceeds ARC: size cache or reduce footprint]

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# Overall hit ratio from raw counters (includes prefetch - rough number only)
awk '/^hits / {h=$3} /^misses / {m=$3} END {printf "hit ratio: %.2f%%\n", h/(h+m)*100}' /proc/spl/kstat/zfs/arcstats

# Demand-only hit ratio, data vs metadata split (the number that matters)
awk '
  /^demand_data_hits/     {ddh=$3}
  /^demand_data_misses/   {ddm=$3}
  /^demand_metadata_hits/ {dmh=$3}
  /^demand_metadata_misses/{dmm=$3}
  END {
    printf "demand data hit ratio:     %.2f%%\n", ddh/(ddh+ddm)*100
    printf "demand metadata hit ratio: %.2f%%\n", dmh/(dmh+dmm)*100
    printf "demand overall hit ratio:  %.2f%%\n", (ddh+dmh)/(ddh+ddm+dmh+dmm)*100
  }' /proc/spl/kstat/zfs/arcstats

# ARC size vs target vs max
awk '/^size / {s=$3} /^c / {c=$3} /^c_max / {cm=$3} /^c_min / {cn=$3}
  END {printf "size=%d c(target)=%d c_max=%d c_min=%d\n", s, c, cm, cn}' /proc/spl/kstat/zfs/arcstats

# Is the ARC being told not to grow? Is memory throttling active?
grep -E '^(arc_no_grow|memory_throttle_count)' /proc/spl/kstat/zfs/arcstats

# Human-readable live view (ships with OpenZFS; renamed zarcstat in 2.4.0)
arcstat 1 10

# Is a scrub or resilver running right now?
zpool status | grep -A3 'scan:'

# System memory context - remember ARC shows as "used", not "available"
grep -E 'MemTotal|MemAvailable' /proc/meminfo

# Who is consuming memory?
ps aux --sort=-%mem | head -10

Two interpretation notes. First, on Linux the ARC lives outside the kernel page cache, so free reports ARC memory as used and the available column understates what the system can reclaim. Do not conclude the machine is out of memory from free alone. Second, if memory_throttle_count is incrementing, ZFS is actively throttling I/O due to memory pressure; that moves this from cache tuning to memory incident.

How to diagnose it

1. Rule out the benign explanations first. Check uptime and pool import time. An ARC warm for 20 minutes will have a terrible hit ratio; that is how caches work. Then check zpool status for an active scrub or resilver. Both walk the entire block tree and destroy the hit ratio for their duration. If either applies, trend the number and re-evaluate after the window.

2. Compute the demand-only ratio, split data vs metadata. The aggregate includes prefetch and will mislead you. If the demand ratio is healthy, your alert was keyed on the aggregate and the threshold needs fixing, not the system.

3. Check the split. If the demand metadata ratio is high but the demand data ratio is low, the system finds files efficiently but re-reads data blocks from disk on every access. The data working set exceeds the data portion of the ARC. Under metadata-heavy workloads (directory traversals, rsync, many small files), metadata can dominate the ARC and crowd data out. The old metadata-limit tunables (zfs_arc_meta_limit, zfs_arc_meta_strategy and friends) were removed in OpenZFS 2.2.0 and replaced by a single zfs_arc_meta_balance parameter, so older tuning guides referencing those knobs do not apply on current releases.

4. Compare ARC size to its ceiling. If size is at or near c_max and the demand data ratio is declining over weeks, the working set outgrew the cache. That is a capacity decision, not a fault. If size is well below c_max and c (the target) has also been pushed down, something is pressuring the ARC. Check arc_no_grow: when set, the memory subsystem has told the ARC not to grow, which explains a pinned ARC with a low hit ratio.

5. If the ARC shrank, find the memory consumer. Use ps aux --sort=-%mem, MemAvailable in /proc/meminfo, and swap usage. The classic pattern is a new application or container workload deployed on the host without adjusting zfs_arc_max. Check dmesg for OOM killer activity; on Linux the OOM killer can fire before the ARC finishes shrinking, because ARC reclaim is not instantaneous. OpenZFS 2.3.0 also changed shrinker behavior (zfs_arc_shrinker_limit default went from 10000 to 0) and there are operator reports of the ARC collapsing far more aggressively under modest memory pressure after upgrading to 2.3.x. If your hit ratio dropped right after a 2.3.x upgrade and ARC size fell toward c_min, check that history before blaming the workload.

6. Correlate with latency and disk I/O to confirm impact. A low hit ratio that does not raise read latency or read IOPS is not hurting anyone (sequential streaming is the usual case). The actionable combination is: demand hit ratio down, ARC size flat or shrinking, disk read IOPS up, read latency up. zpool iostat -l 1 gives the latency side, zpool iostat -v 1 the per-vdev read side.

7. Establish what the baseline actually was. Pull hit ratio history for the last 30 to 90 days. A slow decline over weeks is working-set growth. A step change on a specific day is a workload, config, or version change. A sawtooth aligned with scrub schedules is instrumentation noise you should stop alerting on.

Metrics and signals to monitor

Fixes

Cold cache or scrub-induced dip

No fix; do not page anyone. Suppress or downgrade hit-ratio alerts during boot, import, and scrub windows. With L2ARC on OpenZFS 2.0 or later, persistent L2ARC (l2arc_rebuild_enabled=1, the default) rebuilds the index on import and shortens warmup, but does not eliminate it.

ARC shrank under memory pressure

Set an explicit ceiling so the ARC and applications do not fight:

# Immediate, non-persistent: cap ARC at e.g. 32 GiB. The ARC shrinks to the
# new limit right away, so expect an eviction burst and a disk-read spike.
# Do this during low load.
echo 34359738368 > /sys/module/zfs/parameters/zfs_arc_max

# Persistent across reboots
echo 'options zfs zfs_arc_max=34359738368' >> /etc/modprobe.d/zfs.conf

The sizing tradeoff is real: cap too low and you re-create the low hit ratio you were fixing; cap too high and the OOM killer may kill applications before the ARC finishes shrinking. A common starting point on a shared host is to leave 20 to 25 percent of RAM for the OS and applications, and to size zfs_arc_max explicitly on any machine running memory-hungry workloads alongside ZFS. If a runaway process caused the pressure, fix that first; the cap is a guardrail, not a remedy for a leak. For database datasets that keep their own buffer cache, consider primarycache=metadata so ZFS stops double-caching data blocks the database already caches.

Working set outgrew the ARC

Three options, in order of preference:

1. Add RAM and raise zfs_arc_max. The only true fix for a working-set problem.
2. Reduce the working set. Archive cold datasets, split hot and cold data onto different pools, or fix applications scanning more data than they need (unindexed queries, full-directory walks).
3. Add L2ARC if reads are latency-sensitive and you have spare fast SSD capacity. Validate it: L2ARC only caches blocks evicted from the ARC, consumes ARC RAM for its headers (roughly 70 bytes per cached block), and is a net negative if its hit ratio stays low. If l2_hits / (l2_hits + l2_misses) sits under about 10% sustained, remove it and give the header RAM back to the ARC.

The workload is cache-hostile by design

Sequential streaming, backups, and random reads over datasets far larger than RAM will never cache well, and no tuning changes that. Confirm the demand hit ratio is not causing user-visible latency, then set the alert threshold to the workload’s real baseline instead of a generic 80 or 95 percent. A threshold that fires every backup window is training the team to ignore it.

Metadata crowding out data

On OpenZFS 2.2.0 and later, metadata-vs-data balance is governed by zfs_arc_meta_balance (default 500, meaning metadata ghost hits carry 5x the weight of data ghost hits). If a metadata-heavy workload is evicting hot data, lowering this value biases the ARC toward retaining data. This is a workload-specific tune; change it with the demand data/metadata split in front of you, and revert if the split does not improve.

Prevention

• Alert on deviation from baseline, not absolute values. A file server and a random-read database have completely different healthy hit ratios. Build the baseline per pool and per workload over at least a few weeks, including scrub windows.
• Alert on the demand ratio, split by data and metadata. The aggregate number causes false pages and hides real regressions.
• Set zfs_arc_max explicitly on every shared host. Do not rely on defaults when ZFS shares a machine with databases, JVMs, or containers. Keep 20 to 25 percent of RAM outside the ARC.
• Correlate before alerting. The actionable page is demand hit ratio down + ARC size shrinking or at c_max + read latency up. A bare hit-ratio threshold false-fires on every cold boot, scrub, and backup.
• Track ARC size, arc_no_grow, and memory_throttle_count as leading indicators. The hit ratio is the last thing to move in a memory-pressure cascade; size and throttle counters move first.
• Re-baseline after upgrades. ARC eviction and shrinker behavior has changed materially across OpenZFS releases (2.2.0 eviction rewrite, 2.3.0 shrinker and default-size changes). A hit-ratio shift that starts on upgrade day is a version behavior question before it is a workload question.

How Netdata helps

• Netdata collects /proc/spl/kstat/zfs/arcstats continuously, so demand data/metadata hit ratios, ARC size vs target vs max, and eviction counters are time series instead of point-in-time awk snapshots during an incident.
• Per-second granularity catches short memory-pressure episodes a 5-minute scrape smooths over: ARC size dips, memory_throttle_count increments, and the read-IOPS spike that follows.
• Correlating ARC charts with system memory (MemAvailable, swap), per-disk I/O, and application memory on the same dashboard turns “hit ratio is low” into “the new container workload squeezed the ARC from 60 GiB to 8 GiB starting Tuesday.”
• Historical retention lets you build the per-workload baseline this signal requires and alert on deviation rather than a generic threshold.
• Anomaly detection on the demand hit ratio flags step changes (deploys, upgrades, workload shifts) without hand-tuned thresholds per pool.

Related guides

• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• ZFS capacity cliff: why the pool falls off a performance edge near 80-90% full
• ZFS No space left on device: ENOSPC, the slop reserve, and the pool you cannot delete from
• ZFS pool ONLINE with non-zero errors: why zpool status -x lies
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS pool DEGRADED: redundancy lost and one failure from data loss
• ZFS pool FAULTED: when the pool can no longer serve I/O
• ZFS permanent errors have been detected in the following files: recovering from data loss