ZFS L2ARC ineffective: a cache that burns SSD endurance for nothing

You added an SSD as an L2ARC device expecting faster reads. Months later, read latency has not moved, the SSD’s wear indicator is climbing, and the ARC is smaller than it should be. The L2ARC is being written to constantly and read from almost never. You are paying for the cache in RAM and drive endurance and getting nothing back.

This is one of the most common ZFS mistakes: L2ARC is assumed to be beneficial by default, and almost nobody validates it after adding it. Many workloads get zero benefit from it, and for those workloads the device is a net negative: it consumes ARC memory for its index headers and burns SSD write endurance caching blocks that are never read again.

This article covers how L2ARC actually works, how to measure whether yours is earning its keep, and how to remove it cleanly when it is not.

What this means

L2ARC is not a general-purpose read cache. Three design properties explain most “ineffective L2ARC” situations:

It focuses on ARC candidates, not first reads. L2ARC does not cache data on first read. In the traditional mode (total cache capacity below twice arc_c_max), a block must be read into the ARC and evicted before L2ARC writes it. In current OpenZFS, a sufficiently large L2ARC (at least twice arc_c_max) can use persistent markers to cache content before eviction, but still follows ARC recency and feed-rate limits. If your working set fits in the ARC, there are few useful candidates, and the L2ARC sits mostly empty. If your workload is streaming (backups, media, analytics scans), blocks are read once and never again, so caching them is pure waste.

It only helps on the second access. The first read of any block always goes to disk. The second read only hits L2ARC if the block was evicted from the ARC in between. Workloads without that revisit pattern see no hits at all.

Its index lives in your RAM. Every block cached on L2ARC costs ARC memory for an L2-only header; the exact size depends on OpenZFS release and platform ABI, with operator estimates historically ranging from about 70 to 170 bytes. Current OpenZFS allocates an L2-only prefix of the ARC header and reports the actual total in l2_hdr_size, so measure it rather than assuming the historical constant. A large L2ARC device can consume gigabytes of ARC just for the index, and that RAM comes directly out of what the ARC could use for actual cached data. A low-hit L2ARC makes the ARC smaller and less effective while providing nothing in return: a double loss.

flowchart TD
  A[Application read] --> B{In ARC?}
  B -->|yes| C[Served from RAM]
  B -->|no| D{In L2ARC?}
  D -->|yes| E[Served from SSD]
  D -->|no| F[Read from pool disks]
  F --> G[Block enters ARC]
  G --> H[ARC eviction under pressure]
  H --> I[L2ARC feed writes block to SSD]
  I --> J[Header costs ARC RAM per block]
  J -.->|never re-read| K[SSD endurance spent, zero hits]

Without persistent L2ARC (OpenZFS 2.0+, enabled by default via l2arc_rebuild_enabled=1), the L2ARC is also completely cold after every reboot. Even with persistence, the index is rebuilt on pool import and the cache takes time to become useful again. If you reboot frequently, you may pay the fill cost over and over without ever reaching steady state.

Common causes

Quick checks

All of these are read-only and safe to run on a production system.

# Pull the L2ARC-relevant counters from arcstats
awk '/^l2_hits|^l2_misses|^l2_size|^l2_asize|^l2_hdr_size|^l2_read_bytes|^l2_write_bytes|^l2_io_error|^l2_cksum_bad/ {print $1, $3}' /proc/spl/kstat/zfs/arcstats

# Compute the L2ARC hit ratio
awk '/^l2_hits/ {h=$3} /^l2_misses/ {m=$3} END {if (h+m>0) printf "L2ARC hit ratio: %.2f%%\n", h/(h+m)*100; else print "No L2ARC activity"}' /proc/spl/kstat/zfs/arcstats

# Compare header RAM cost against total ARC size
awk '/^l2_hdr_size/ {hdr=$3} /^size/ {s=$3} END {printf "l2_hdr_size: %.1f MiB, ARC size: %.1f GiB, headers = %.2f%% of ARC\n", hdr/1048576, s/1073741824, hdr/s*100}' /proc/spl/kstat/zfs/arcstats

# Check whether L2ARC writes vastly exceed reads (burning endurance for nothing)
awk '/^l2_read_bytes/ {r=$3} /^l2_write_bytes/ {w=$3} END {printf "l2_read_bytes: %.1f GiB, l2_write_bytes: %.1f GiB, read:write ratio 1:%.1f\n", r/1073741824, w/1073741824, (r>0?w/r:w)}' /proc/spl/kstat/zfs/arcstats

# Confirm the cache device state and any device-level errors
zpool status -v

# Check SSD wear on the cache device (adjust device name)
smartctl -A /dev/sdX

The arcstats counters are cumulative since boot, so a hit ratio computed from them hides recent behavior. Take two samples minutes or hours apart and compute the ratio over the delta to see what the L2ARC is doing now, not what it did last month.

How to diagnose it

1. Establish the interval hit ratio. Sample l2_hits and l2_misses twice, at least an hour apart during representative load, and compute delta_hits / (delta_hits + delta_misses). Sustained ratios below roughly 10 to 30 percent mean the L2ARC is not earning its overhead. The decision threshold is workload-dependent, but if you are consistently at the low end of that range, the cache is decorative.

2. Check the read-to-write balance. Compare l2_read_bytes to l2_write_bytes over the same interval. Writes massively exceeding reads is the signature of the L2ARC burnout pattern: the feed thread is streaming data onto the SSD that nobody reads back, and every one of those bytes consumes drive endurance.

3. Quantify the RAM cost. Divide l2_hdr_size by the ARC size. Header overhead should stay well under a tenth of the ARC. If headers are consuming 5 to 10 percent of the ARC while the hit ratio is low, the L2ARC is a net negative: removing it gives that RAM back to the ARC, which is a strictly better cache.

4. Check whether the workload can ever benefit. Ask whether the same blocks are re-read after ARC eviction. Backups, log archival, video streaming, and analytics scans are read-once workloads. Random re-reads of a working set modestly larger than RAM are where L2ARC helps. If your workload is the former, no tuning will fix it.

5. Factor in reboot frequency. If the system reboots often and you are not on OpenZFS 2.0 or later (or persistence is disabled), the cache restarts cold every boot and may never warm up before the next reboot. Check uptime against your hit ratio trend.

6. Rule out the opposite problem first. Before blaming L2ARC for slow reads, confirm the ARC itself is healthy. A low ARC hit ratio caused by a shrunken or undersized ARC is a different problem with a different fix. See the related guides below.

Metrics and signals to monitor

Fixes

Remove the L2ARC device

If the interval hit ratio is sustained below roughly 10 to 30 percent and headers are consuming meaningful ARC, the correct fix is removal:

# Identify the cache device first with zpool status, then remove it
zpool remove <pool> <cache-device>

This is safe: L2ARC contains only cached copies, never unique data. Removal is orderly, and the header RAM returns to the ARC immediately. Reads that would have hit L2ARC fall back to the pool disks, which for a low-hit cache is barely measurable. It is not disruptive, but a maintenance window costs you nothing if you want to be conservative.

After removal, watch the ARC hit ratio for a few days. It should improve as the freed header RAM is put to use.

Keep it, but only if the numbers justify it

If the interval hit ratio is comfortably above 30 percent and l2_hdr_size is a small fraction of the ARC, the L2ARC is working. Leave it alone. Keep trending the hit ratio and SSD wear, because workload drift can turn a good L2ARC into a bad one over time.

Reduce the RAM tax and write burn if you keep it

• Right-size the device. Header cost scales with cached blocks, so a smaller cache device means fewer headers. Estimate the header RAM from device size and your dataset recordsize before you ever attach the device.
• Restrict what gets cached. The secondarycache dataset property controls what L2ARC stores per dataset: all, metadata, or none. Setting secondarycache=none on backup, archive, or other scan-heavy datasets stops one-time-read data from ever entering the feed, cutting both SSD wear and header overhead while keeping the cache for workloads that benefit.
• Throttle the feed. L2ARC write speed is bounded by the l2arc_write_max and l2arc_write_boost tunables. Lowering the write rate reduces endurance consumption at the cost of slower cache warmup. The default rose from 8 MiB to 32 MiB per second in OpenZFS 2.2.7; check the installed release before assuming a value.
• Enable persistence if you reboot. On OpenZFS 2.0 and later, l2arc_rebuild_enabled=1 (the default) rebuilds the L2ARC index on pool import so the cache survives reboots. On older versions, every reboot is a cold start.

Do not confuse this with an ARC problem

If your real symptom is a low ARC hit ratio, adding or keeping L2ARC is usually the wrong response. The better levers are raising zfs_arc_max if the ARC is artificially capped, adding RAM, or reducing memory competition from applications. L2ARC is a second-tier cache with a RAM tax; the first tier is almost always where the win is.

Prevention

• Validate before and after adding L2ARC. Estimate header RAM cost up front, and set a review checkpoint after deployment: if the interval hit ratio is below your threshold after a few weeks of representative load, remove it.
• Trend, do not snapshot. Point-in-time zpool status and cumulative arcstats counters hide drift. Export l2_hits, l2_misses, l2_hdr_size, l2_read_bytes, and l2_write_bytes to a time-series system so you can compute interval ratios and spot workload changes.
• Monitor cache device wear like a SLOG. L2ARC and SLOG devices take concentrated write load. Track SSD SMART or NVMe wear indicators and plan replacement before exhaustion, same as you would for a log device.
• Set secondarycache deliberately on scan-heavy datasets rather than letting one-time-read data flow into the cache by default.
• Cap the ARC explicitly. An uncapped ARC plus L2ARC headers plus applications is how you end up in memory-pressure incidents. See the related guides on zfs_arc_max and ARC memory behavior.

How Netdata helps

• Netdata collects the ZFS ARC statistics from /proc/spl/kstat/zfs/arcstats per second, including l2_hits, l2_misses, l2_hdr_size, l2_read_bytes, and l2_write_bytes, so you see the L2ARC hit ratio and read/write balance as trends rather than cumulative snapshots.
• Correlating L2ARC hits against ARC hits and pool disk read IOPS shows whether the L2ARC is actually absorbing reads that would otherwise hit the data vdevs, or whether disk reads are unchanged while the SSD fills.
• Charting l2_hdr_size next to ARC size makes the RAM tax visible: you can see header growth coinciding with ARC data hit ratio decline.
• Per-second L2ARC write rates combined with disk-level metrics and device wear tracking let you quantify endurance spend per day and project SSD lifetime under the current feed rate.
• Anomaly detection on the L2ARC hit ratio catches workload drift, such as a new batch job that turned a healthy cache into a write-only endurance sink, without manual threshold babysitting.

Related guides

• ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM
• ZFS zfs_arc_max: capping the ARC without starving read performance
• ZFS ARC using all memory: the Linux default that eats your RAM
• ZFS ARC shrinking below c_max: reading memory pressure before latency hits
• ZFS ARC and the OOM killer: applications killed while the cache will not shrink fast enough