You added an SSD as an L2ARC device expecting faster reads. Months later, read latency has not moved, the SSD’s wear indicator is climbing, and the ARC is smaller than it should be. The L2ARC is being written to constantly and read from almost never. You are paying for the cache in RAM and drive endurance and getting nothing back.

This is one of the most common ZFS mistakes: L2ARC is assumed to be beneficial by default, and almost nobody validates it after adding it. Many workloads get zero benefit from it, and for those workloads the device is a net negative: it consumes ARC memory for its index headers and burns SSD write endurance caching blocks that are never read again.

This article covers how L2ARC actually works, how to measure whether yours is earning its keep, and how to remove it cleanly when it is not.

What this means

L2ARC is not a general-purpose read cache. Three design properties explain most “ineffective L2ARC” situations:

It only caches ARC evictions. L2ARC does not cache data on first read. A block must be read into the ARC, later evicted from the ARC, and only then is it eligible to be written to L2ARC. If your working set fits in the ARC, nothing ever gets evicted, and the L2ARC sits mostly empty. If your workload is streaming (backups, media, analytics scans), blocks are read once and never again, so caching the eviction is pure waste.

It only helps on the second access. The first read of any block always goes to disk. The second read only hits L2ARC if the block was evicted from the ARC in between. Workloads without that revisit pattern see no hits at all.

Its index lives in your RAM. Every block cached on L2ARC costs ARC memory for its header, roughly 70 bytes per block. A large L2ARC device can consume gigabytes of ARC just for the index, and that RAM comes directly out of what the ARC could use for actual cached data. A low-hit L2ARC makes the ARC smaller and less effective while providing nothing in return: a double loss.

flowchart TD
  A[Application read] --> B{In ARC?}
  B -->|yes| C[Served from RAM]
  B -->|no| D{In L2ARC?}
  D -->|yes| E[Served from SSD]
  D -->|no| F[Read from pool disks]
  F --> G[Block enters ARC]
  G --> H[ARC eviction under pressure]
  H --> I[L2ARC feed writes block to SSD]
  I --> J[Header costs ARC RAM per block]
  J -.->|never re-read| K[SSD endurance spent, zero hits]

Without persistent L2ARC (OpenZFS 2.0+, enabled by default via l2arc_rebuild_enabled=1), the L2ARC is also completely cold after every reboot. Even with persistence, the index is rebuilt on pool import and the cache takes time to become useful again. If you reboot frequently, you may pay the fill cost over and over without ever reaching steady state.

Common causes

CauseWhat it looks likeFirst thing to check
Working set fits in ARCL2ARC nearly empty, near-zero hits, ARC hit ratio already highARC hit ratio and eviction rate in arcstats
Streaming or scan-heavy workloadHigh l2_write_bytes, near-zero l2_read_bytes, blocks read once and never againCompare l2_read_bytes vs l2_write_bytes over time
Working set far larger than ARC plus L2ARCHigh miss rates everywhere, L2ARC churns constantlyl2_hits vs l2_misses ratio
Frequent reboots without persistenceL2ARC cold after every boot, hit ratio never climbsUptime vs L2ARC hit ratio trend
L2ARC headers eating ARC RAMl2_hdr_size is a large fraction of ARC size, ARC data hit ratio degradedl2_hdr_size vs ARC size in arcstats
Cache device too small or too slow to matterL2ARC fills and wraps without capturing the hot setl2_size and l2_asize vs workload working set

Quick checks

All of these are read-only and safe to run on a production system.

# Pull the L2ARC-relevant counters from arcstats
awk '/^l2_hits|^l2_misses|^l2_size|^l2_asize|^l2_hdr_size|^l2_read_bytes|^l2_write_bytes|^l2_io_error|^l2_cksum_bad/ {print $1, $3}' /proc/spl/kstat/zfs/arcstats

# Compute the L2ARC hit ratio
awk '/^l2_hits/ {h=$3} /^l2_misses/ {m=$3} END {if (h+m>0) printf "L2ARC hit ratio: %.2f%%\n", h/(h+m)*100; else print "No L2ARC activity"}' /proc/spl/kstat/zfs/arcstats

# Compare header RAM cost against total ARC size
awk '/^l2_hdr_size/ {hdr=$3} /^size/ {s=$3} END {printf "l2_hdr_size: %.1f MiB, ARC size: %.1f GiB, headers = %.2f%% of ARC\n", hdr/1048576, s/1073741824, hdr/s*100}' /proc/spl/kstat/zfs/arcstats

# Check whether L2ARC writes vastly exceed reads (burning endurance for nothing)
awk '/^l2_read_bytes/ {r=$3} /^l2_write_bytes/ {w=$3} END {printf "l2_read_bytes: %.1f GiB, l2_write_bytes: %.1f GiB, read:write ratio 1:%.1f\n", r/1073741824, w/1073741824, (r>0?w/r:w)}' /proc/spl/kstat/zfs/arcstats

# Confirm the cache device state and any device-level errors
zpool status -v

# Check SSD wear on the cache device (adjust device name)
smartctl -A /dev/sdX

The arcstats counters are cumulative since boot, so a hit ratio computed from them hides recent behavior. Take two samples minutes or hours apart and compute the ratio over the delta to see what the L2ARC is doing now, not what it did last month.

How to diagnose it

  1. Establish the interval hit ratio. Sample l2_hits and l2_misses twice, at least an hour apart during representative load, and compute delta_hits / (delta_hits + delta_misses). Sustained ratios below roughly 10 to 30 percent mean the L2ARC is not earning its overhead. The decision threshold is workload-dependent, but if you are consistently at the low end of that range, the cache is decorative.

  2. Check the read-to-write balance. Compare l2_read_bytes to l2_write_bytes over the same interval. Writes massively exceeding reads is the signature of the L2ARC burnout pattern: the feed thread is streaming data onto the SSD that nobody reads back, and every one of those bytes consumes drive endurance.

  3. Quantify the RAM cost. Divide l2_hdr_size by the ARC size. Header overhead should stay well under a tenth of the ARC. If headers are consuming 5 to 10 percent of the ARC while the hit ratio is low, the L2ARC is a net negative: removing it gives that RAM back to the ARC, which is a strictly better cache.

  4. Check whether the workload can ever benefit. Ask whether the same blocks are re-read after ARC eviction. Backups, log archival, video streaming, and analytics scans are read-once workloads. Random re-reads of a working set modestly larger than RAM are where L2ARC helps. If your workload is the former, no tuning will fix it.

  5. Factor in reboot frequency. If the system reboots often and you are not on OpenZFS 2.0 or later (or persistence is disabled), the cache restarts cold every boot and may never warm up before the next reboot. Check uptime against your hit ratio trend.

  6. Rule out the opposite problem first. Before blaming L2ARC for slow reads, confirm the ARC itself is healthy. A low ARC hit ratio caused by a shrunken or undersized ARC is a different problem with a different fix. See the related guides below.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
l2_hits / (l2_hits + l2_misses)Direct measure of L2ARC effectivenessSustained below 10-30% on interval deltas
l2_write_bytes vs l2_read_bytesEndurance spend vs benefit deliveredWrites far exceed reads over long intervals
l2_hdr_size vs ARC sizeRAM tax the cache imposes on the ARCHeaders approaching a tenth of ARC with low hits
l2_size / l2_asizeHow much of the device is actually usedDevice full and churning with low hit ratio
l2_io_error, l2_cksum_badCache device health (tolerated, but degrades value)Any sustained increment
SSD SMART wear indicatorsEndurance consumed by L2ARC feed writesWear climbing faster than the rest of your fleet
ARC hit ratioThe cache that actually mattersDropping while l2_hdr_size grows

Fixes

Remove the L2ARC device

If the interval hit ratio is sustained below roughly 10 to 30 percent and headers are consuming meaningful ARC, the correct fix is removal:

# Identify the cache device first with zpool status, then remove it
zpool remove <pool> <cache-device>

This is safe: L2ARC contains only cached copies, never unique data. Removal is orderly, and the header RAM returns to the ARC immediately. Reads that would have hit L2ARC fall back to the pool disks, which for a low-hit cache is barely measurable. It is not disruptive, but a maintenance window costs you nothing if you want to be conservative.

After removal, watch the ARC hit ratio for a few days. It should improve as the freed header RAM is put to use.

Keep it, but only if the numbers justify it

If the interval hit ratio is comfortably above 30 percent and l2_hdr_size is a small fraction of the ARC, the L2ARC is working. Leave it alone. Keep trending the hit ratio and SSD wear, because workload drift can turn a good L2ARC into a bad one over time.

Reduce the RAM tax and write burn if you keep it

  • Right-size the device. Header cost scales with cached blocks, so a smaller cache device means fewer headers. Estimate the header RAM from device size and your dataset recordsize before you ever attach the device.
  • Restrict what gets cached. The secondarycache dataset property controls what L2ARC stores per dataset: all, metadata, or none. Setting secondarycache=none on backup, archive, or other scan-heavy datasets stops one-time-read data from ever entering the feed, cutting both SSD wear and header overhead while keeping the cache for workloads that benefit.
  • Throttle the feed. L2ARC write speed is bounded by the l2arc_write_max and l2arc_write_boost tunables. Lowering the write rate reduces endurance consumption at the cost of slower cache warmup.
  • Enable persistence if you reboot. On OpenZFS 2.0 and later, l2arc_rebuild_enabled=1 (the default) rebuilds the L2ARC index on pool import so the cache survives reboots. On older versions, every reboot is a cold start.

Do not confuse this with an ARC problem

If your real symptom is a low ARC hit ratio, adding or keeping L2ARC is usually the wrong response. The better levers are raising zfs_arc_max if the ARC is artificially capped, adding RAM, or reducing memory competition from applications. L2ARC is a second-tier cache with a RAM tax; the first tier is almost always where the win is.

Prevention

  • Validate before and after adding L2ARC. Estimate header RAM cost up front, and set a review checkpoint after deployment: if the interval hit ratio is below your threshold after a few weeks of representative load, remove it.
  • Trend, do not snapshot. Point-in-time zpool status and cumulative arcstats counters hide drift. Export l2_hits, l2_misses, l2_hdr_size, l2_read_bytes, and l2_write_bytes to a time-series system so you can compute interval ratios and spot workload changes.
  • Monitor cache device wear like a SLOG. L2ARC and SLOG devices take concentrated write load. Track SSD SMART or NVMe wear indicators and plan replacement before exhaustion, same as you would for a log device.
  • Set secondarycache deliberately on scan-heavy datasets rather than letting one-time-read data flow into the cache by default.
  • Cap the ARC explicitly. An uncapped ARC plus L2ARC headers plus applications is how you end up in memory-pressure incidents. See the related guides on zfs_arc_max and ARC memory behavior.

How Netdata helps

  • Netdata collects the ZFS ARC statistics from /proc/spl/kstat/zfs/arcstats per second, including l2_hits, l2_misses, l2_hdr_size, l2_read_bytes, and l2_write_bytes, so you see the L2ARC hit ratio and read/write balance as trends rather than cumulative snapshots.
  • Correlating L2ARC hits against ARC hits and pool disk read IOPS shows whether the L2ARC is actually absorbing reads that would otherwise hit the data vdevs, or whether disk reads are unchanged while the SSD fills.
  • Charting l2_hdr_size next to ARC size makes the RAM tax visible: you can see header growth coinciding with ARC data hit ratio decline.
  • Per-second L2ARC write rates combined with disk-level metrics and device wear tracking let you quantify endurance spend per day and project SSD lifetime under the current feed rate.
  • Anomaly detection on the L2ARC hit ratio catches workload drift, such as a new batch job that turned a healthy cache into a write-only endurance sink, without manual threshold babysitting.