ZFS dedup memory exhaustion: when the DDT outgrows ARC and the pool crawls

The pool is ONLINE. zpool status -x says everything is healthy. Disk latencies look mediocre but not dead. Yet every write takes tens to hundreds of milliseconds, reads that used to come from cache now hit disk, and application latency is uniformly awful in both directions. Nothing is DEGRADED, no scrub is running, and capacity looks unremarkable.

If dedup=on is set anywhere on that pool, this is the signature of one of the most expensive mistakes in ZFS operations: the deduplication table (DDT) has outgrown the ARC, and the pool is doing a random disk read for nearly every write just to answer the question “have I seen this block before?”

This article covers how to confirm that diagnosis quickly, why the failure mode is so total, and the genuinely bad news about the way out.

What this means

ZFS dedup keeps a table, the DDT, with one entry per unique block. Each entry costs roughly 320 bytes of RAM. Every write must consult the DDT to decide whether the block is new or a duplicate, and every delete or overwrite must update reference counts in it. That table must live in memory to be usable, and it lives inside the ARC.

The failure is a two-stage collapse:

1. The DDT grows past what the ARC can hold. Now DDT lookups miss and become synchronous disk reads. Write throughput drops from “speed of your vdevs” to “speed of random reads on your vdevs,” which on spinning disks is a collapse of two to three orders of magnitude.
2. The DDT pages that are in ARC act like pinned metadata. They crowd out actual data cache. Read hit ratios crater, so reads now also go to disk, competing with the DDT reads the writes are generating.

Reads and writes both die, from one cause. This is why operators misdiagnose it as a disk or controller problem: the entire storage stack is slow, uniformly.

flowchart TD
  A[dedup=on, DDT grows] --> B[DDT exceeds ARC capacity]
  B --> C[DDT lookups miss to disk]
  B --> D[DDT crowds out data cache]
  C --> E[write latency collapses to random-read speed]
  D --> F[ARC hit ratio craters]
  F --> G[reads hit disk too]
  E --> H[pool-wide I/O saturation]
  G --> H
  H --> I[uniform slowness, pool still ONLINE]

The pool stays ONLINE the entire time. There are no error counters, no deadman events, no checksum failures. Every health signal you probably alert on stays green.

Common causes

Quick checks

These are all read-only and safe to run on a live pool. One caution: zdb -S performs a simulation pass that reads the pool’s data blocks, and it is I/O intensive on a large pool. Run it during a low-traffic window if the pool is already struggling.

# Is dedup actually on, and where?
zfs get -r dedup <pool>

# DDT statistics: entry count, on-disk and in-core size, dedup ratio
zpool status -D <pool>

# Simulate DDT size and dedup ratio (I/O intensive; use off-peak)
zdb -S <pool>

# ARC state: current size, target, cap
grep -E '^(size|c|c_max|c_min|hits|misses)' /proc/spl/kstat/zfs/arcstats

# Metadata pressure inside the ARC
grep -E '^(arc_meta_used|arc_meta_limit)' /proc/spl/kstat/zfs/arcstats

# Latency and queue evidence of the collapse
zpool iostat -l 5
zpool iostat -q -v 5

Interpretation shortcuts:

• zpool status -D showing an in-core DDT size comparable to or larger than your ARC size is near-certain confirmation.
• ARC metadata usage dominating the cache while the data hit ratio collapses is the ARC-side fingerprint. Metadata hit ratio stays high (the DDT is metadata and gets priority treatment in practice), data hit ratio goes to near zero.
• A dedup ratio near 1.0x means you are paying the entire DDT cost for essentially no space savings. The table grows with every unique block written whether or not dedup is helping.

How to diagnose it

1. Confirm dedup is enabled. zfs get -r dedup <pool>. If every dataset says off, this is not your problem; go look at TXG sync pressure or device latency instead.
2. Measure the DDT. zpool status -D <pool> for the current table. zdb -S <pool> for the simulation view of entries and projected sizes. Multiply the unique entry count by roughly 320 bytes to get the RAM requirement.
3. Compare against the ARC. Pull size, c, and c_max from /proc/spl/kstat/zfs/arcstats. If the DDT in-core requirement approaches or exceeds the ARC target, you have the smoking gun.
4. Verify the ARC split. Metadata usage at or near the metadata limit while demand data hit ratio collapses confirms the cache eviction half of the failure. Watch hits and misses over an interval with arcstat 1 rather than lifetime counters.
5. Rule out competing explanations. Check zpool iostat -v for a single slow device, check zpool status for an active scrub or resilver, and check TXG sync times in /proc/spl/kstat/zfs/<pool>/txgs (stime field). DDT exhaustion produces uniformly bad latency across all vdevs with no single culprit device, which is itself diagnostic.
6. Check the dedup ratio. If the ratio is below roughly 2x, dedup was a bad trade for this workload regardless of the memory situation. That fact matters for the remediation decision below.

Metrics and signals to monitor

Only monitor these on pools where dedup=on. Most pools should not use dedup, and DDT metrics on a non-dedup pool are noise.

Severity guidance: DDT exceeding 25% of ARC is a ticket (dedup is displacing useful cache), dedup ratio approaching 1.0x is a ticket (paying cost for no benefit), and a DDT that no longer fits in ARC with collapsed hit ratios is a page, because the pool is effectively down for performance purposes.

Fixes

There is no quick fix. This is the part where honesty matters more than comfort.

The only real fix: replicate out and rebuild without dedup

zfs set dedup=off <dataset> stops the bleeding only for new writes. Every block already written stays deduplicated, the DDT stays in the pool and in memory, and every read or delete of existing data keeps paying DDT costs. Setting dedup=off alone does not recover performance.

The actual remediation is to copy the data to datasets (or a new pool) with dedup never enabled, using zfs send | zfs recv, then destroy the deduplicated originals. Only when the last deduplicated block is freed does the DDT go away. Verify the received data before destroying anything: once you run zfs destroy on the originals, the copies are all you have.

The ugly part: the migration itself is slow, because reading the source data requires DDT lookups on a DDT that does not fit in RAM. Budget for this. A large pool in this state can take days to replicate out. Practical mitigations during the copy:

• Run the send/recv during the lowest-traffic windows you have.
• Reduce competing application I/O so the DDT reads and the replication stream are not fighting for the same disks.
• Add RAM before starting if the hardware allows it. Getting the DDT back inside the ARC converts the migration from random-read-bound to sequential-read-bound. This is the single most effective lever for making the copy tolerable, and it also stabilizes the pool while you work.
• Replicate dataset by dataset so you can retire the worst offenders first and watch DDT pressure decline as originals are destroyed.

Things that do not fix it

• Setting dedup=off: new writes only, as above. Do it anyway to stop growth, but do not expect recovery.
• Adding L2ARC: the DDT must effectively be resident to be useful, and cache tiers do not change the requirement that lookups be cheap. Worse, L2ARC consumes ARC memory for its headers, so on a host already short on RAM it can deepen the pressure it was meant to relieve.
• Raising zfs_arc_max alone: helps only if free RAM exists to grow into and the enlarged ARC can actually contain the whole DDT. If the DDT needs 80 GB and the host has 64 GB, no tunable fixes arithmetic.
• Scrub, resilver, device replacement: the hardware is fine. Do not let a green zpool status and uniform slowness send you down a disk-firmware investigation.

If you are reading this before enabling dedup

Run zdb -S <pool> first. It simulates the DDT and reports the achievable dedup ratio and table size without enabling anything. The decision rule: if the ratio is not comfortably above 2x, or the projected table multiplied by ~320 bytes per entry does not fit in ARC with generous headroom for actual data cache, do not enable dedup. Compression, and for suitable workloads copies/clones, deliver most of the space benefit without a permanent RAM tax.

Prevention

• Size before you enable. zdb -S on a representative dataset, then budget DDT RAM at ~320 bytes per unique entry, with headroom. Recheck as the dataset grows; the DDT grows with unique blocks, not with logical size.
• Treat dedup as a hardware commitment. DDT entries are effectively permanent residents for as long as the deduplicated data exists. The RAM is not reclaimable cache in any meaningful operational sense.
• Alert on the ratio and the table size, not just on symptoms. DDT share of ARC climbing past 25%, or dedup ratio drifting toward 1.0x, are your early warnings while remediation is still cheap.
• Cap and watch the ARC. A properly set zfs_arc_max keeps the accounting honest: see ZFS ARC using all memory and capping the ARC without starving reads. A shrinking ARC under memory pressure can push a previously-fitting DDT over the edge; see ARC shrinking below c_max.
• Prefer not enabling dedup at all unless the workload is a known dedup win (for example, large numbers of genuinely duplicate images) and the RAM budget is deliberate. The default answer for general-purpose pools is no.

How Netdata helps

• ARC internals over time: Netdata collects /proc/spl/kstat/zfs/arcstats continuously, so you can see ARC size, target, hit ratios, and metadata pressure as trends rather than one-off zpool status snapshots. The slow creep of metadata displacing data cache is only visible in a time series.
• Correlation of the collapse: the diagnostic pattern is metadata-high plus data-hit-ratio-low plus latency-up across all vdevs simultaneously. Having ARC stats and pool latency on the same dashboard makes the “one cause, both directions” signature visible in seconds instead of after a disk investigation.
• Memory pressure context: memory_throttle_count and system-level memory alongside ARC size distinguish “DDT too big for ARC” from “ARC being squeezed by something else,” which changes the remediation.
• Baseline drift alerts: dedup ratio and hit-ratio degradation happen over weeks. Alerting on deviation from established baselines catches the trajectory while zfs send | recv is still a weekend job rather than a multi-day incident.

Related guides

• ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM
• ZFS zfs_arc_max: capping the ARC without starving read performance
• ZFS ARC and the OOM killer: applications killed while the cache will not shrink fast enough
• ZFS ARC shrinking below c_max: reading memory pressure before latency hits
• ZFS ARC using all memory: the Linux default that eats your RAM
• ZFS capacity planning: runway estimation before the pool fills
• ZFS dirty data throttling: the write delay that masquerades as slow disks
• How ZFS actually works in production: a mental model for operators