ZFS dirty data throttling: the write delay that masquerades as slow disks

Your applications report write latency spiking from microseconds to tens or hundreds of milliseconds. Throughput falls off a cliff under sustained write load. iostat shows the disks are not busy. SMART is clean. The NVMe drives benchmark fine. Everything points at the storage, and nothing is wrong with the storage.

This is the most misdiagnosed latency source in ZFS: the dirty data write throttle. When dirty (uncommitted) data in RAM crosses a threshold, ZFS deliberately injects artificial delay into every write syscall to slow writers down. If dirty data reaches the hard limit, writes stall completely until the syncing transaction group finishes. The system is working exactly as designed: it protects the pool from memory exhaustion by making applications wait.

The trap is the default sizing. zfs_dirty_data_max defaults to 10% of physical RAM, capped by zfs_dirty_data_max_max (by default, the lesser of 1/4 of RAM and 4 GiB). On a machine with NVMe devices that sustain multiple GB/s of writes, a 4GB buffer fills in seconds. The throttle engages not because the disks are slow but because the buffer is artificially small. Operators see write throughput collapse and replace perfectly good hardware.

What this means

ZFS batches all writes into transaction groups (TXGs), flushed to disk periodically (default every 5 seconds via zfs_txg_timeout). Three TXGs are always in flight: one open and accepting writes, one quiescing, one syncing to disk. Writes accumulate in RAM as dirty data until the syncing TXG commits them.

The dirty data limit exists so a fast writer on a slow pool cannot consume unbounded memory. ZFS manages the buffer with two thresholds:

• Soft throttle at zfs_delay_min_dirty_percent (default 60%) of zfs_dirty_data_max. Above this, each write transaction gets an artificial delay that grows as dirty data climbs toward the max.
• Hard stall at zfs_dirty_data_max itself. At 100%, new writes block until the syncing TXG frees space.

The delay follows min_time = zfs_delay_scale * (dirty - min) / (max - dirty), with zfs_delay_scale defaulting to 500000 (nanoseconds). Because the divisor is (max - dirty), the curve steepens near the limit: delay approaches the 100ms cap as dirty data approaches 100% of max. This is why the symptom is bimodal. Writes are either fast or extremely slow, with little in between.

A related tunable, zfs_dirty_data_sync_percent (default 20%), controls when a TXG sync is kicked off as dirty data accumulates. It should stay below zfs_vdev_async_write_active_min_dirty_percent (default 30%).

flowchart TD
  A[Write latency spikes, disks look idle] --> B{Dirty data above 60% of zfs_dirty_data_max?}
  B -- No --> C[Not the throttle: check TXG stime, vdev latency, ZIL]
  B -- Yes --> D{Dirty data near 100% / hard stall?}
  D -- Yes --> E[Writers blocked until TXG sync completes]
  D -- No --> F[Soft throttle active: per-write artificial delay]
  E --> G{Can the pool flush faster?}
  F --> G
  G -- Disks saturated --> H[Real backend bottleneck: find slow vdev]
  G -- Disks idle, buffer small --> I[Raise zfs_dirty_data_max carefully]

Common causes

Quick checks

All read-only and safe to run during an incident.

# Current dirty data limit and throttle threshold
cat /sys/module/zfs/parameters/zfs_dirty_data_max
cat /sys/module/zfs/parameters/zfs_delay_min_dirty_percent

# Current dirty bytes from recent TXGs
tail -20 /proc/spl/kstat/zfs/<pool>/txgs | awk '$4 ~ /^[0-9]+$/ {sum+=$4} END {print sum+0}' | numfmt --to=iec

# Recent TXG sync times and dirty bytes per TXG
cat /proc/spl/kstat/zfs/<pool>/txgs | tail -20

# Per-vdev latency histograms: is any device actually slow?
zpool iostat -wl <pool> 5

# Per-vdev queue depth: is the backend saturated?
zpool iostat -q -v <pool> 1

# Scrub or resilver active?
zpool status <pool> | grep -A5 "scan:"

To confirm the throttle itself fired, watch dmu_tx_delay and dmu_tx_dirty_delay in /proc/spl/kstat/zfs/dmu_tx. If they increment during the latency spikes, the delay was injected by ZFS, not by the disks.

How to diagnose it

1. Establish the symptom shape. Throttle latency is periodic and bimodal: writes alternate between fast and blocked, roughly on the TXG cycle. Steady uniform slowness points elsewhere.
2. Check dirty data against the limit. Read zfs_dirty_data_max and the ndirty field in the TXG history. Above 60% of max, the soft throttle is active. Near 100%, writers are hard-stalled.
3. Check whether the backend can flush. Read stime (sync duration, nanoseconds) from /proc/spl/kstat/zfs/<pool>/txgs. If stime is well under zfs_txg_timeout (5s default), the pool flushes fine and the buffer is simply too small. If stime regularly exceeds the timeout, the backend is the bottleneck.
4. Rule out a slow device. zpool iostat -wl <pool> 5. One vdev 3x slower than its peers holds up the whole sync. That is a hardware problem, not a tuning problem.
5. Rule out competing I/O. Check for scrub, resilver, or a large snapshot destruction in zpool status. These legitimately extend sync times and can push an otherwise-healthy pool into the throttle.
6. Check other pools. The dirty data limit is global. A busy slow pool (spinning disks, a near-full archive pool) can fill the shared buffer and throttle writes to your fast NVMe pool. Per-pool dirty data limits are an open OpenZFS feature request (#15949), not available through OpenZFS 2.4.4.
7. Decide: small buffer or slow backend. Disks idle + throttle active = buffer too small. Disks saturated + throttle active = backend problem; raising the max will not help and may hurt.

Metrics and signals to monitor

Fixes

Raise zfs_dirty_data_max (only if the backend has headroom)

If disks are idle while the throttle is active, the buffer is undersized. Raise it:

# Takes effect immediately; example: 16GB
echo 17179869184 > /sys/module/zfs/parameters/zfs_dirty_data_max

# Persist across reboots
cat >> /etc/modprobe.d/zfs.conf <<'EOF'
options zfs zfs_dirty_data_max=17179869184
EOF

Critical caveat: the default ceiling zfs_dirty_data_max_max is the lesser of 1/4 of RAM and 4 GiB, and OpenZFS enforces it only when the module initializes. A later runtime change to zfs_dirty_data_max is not re-clamped by that ceiling, but persistence still requires /etc/modprobe.d/zfs.conf. Set both defaults deliberately before loading the module; changing the module requires unmounting all ZFS filesystems and exporting all pools, so plan a maintenance window rather than doing this mid-incident.

Tradeoffs, and they are real:

• Bigger buffer means bigger TXG syncs. Sync time scales with dirty data divided by disk throughput. A 16GB buffer on a pool that writes 500MB/s means roughly 30-second syncs if the buffer fills. Administrative operations (zfs create, clone) can stall for seconds during these syncs.
• More RAM at risk. Dirty data is uncommitted. Larger buffers extend the loss window on a crash and compete with the ARC for memory.
• zfs_delay_min_dirty_percent interacts with the I/O scheduler. Keep it at or above zfs_vdev_async_write_active_max_dirty_percent (both default 60%). If the delay kicks in before the scheduler ramps async writes to full concurrency, you throttle before the disks ever reach full speed.

A sane approach: size the buffer to a few seconds of peak write throughput, verify TXG stime stays well under zfs_txg_timeout under load, and stop there.

Fix the actual backend bottleneck

If stime is over the timeout and queues are deep, the throttle is doing its job. Raising the max just delays the stall and makes each sync longer. Instead: find the slow vdev (zpool iostat -wl), check dmesg for link resets, rule out SMR drives under sustained write load, and check pool capacity and fragmentation (zpool list -o name,cap,frag). Past ~85% capacity, allocator overhead alone can push sync times over the edge.

Separate fast and slow workloads

Because the limit is global, one slow pool degrades write latency on every pool. Until per-pool limits ship, the practical mitigations are separate machines for latency-sensitive pools, or scheduling bulk writes to slow pools in off-hours.

Prevention

• Trend dirty data as a percentage of max, not as an absolute. The percentage tells you how much headroom the write path has before the soft throttle engages.
• Alert on dirty data sustained above 80% of zfs_dirty_data_max with disks not saturated. That combination is the throttle about to bite.
• Baseline TXG stime per pool and alert at 2x zfs_txg_timeout sustained.
• Set tunables deliberately at provisioning time, in /etc/modprobe.d/zfs.conf, including zfs_dirty_data_max_max if you intend to exceed 4GB. Do not discover the module-load-time requirement during an incident.
• Revisit the sizing when hardware changes. A buffer tuned for SATA SSDs is wrong the day you move to NVMe.

How Netdata helps

• Netdata collects ZFS internals including ARC stats and pool I/O, so you can plot dirty data against zfs_dirty_data_max and watch the throttle engage in near real time instead of catching it after users complain.
• Per-second pool latency and throughput let you see the bimodal stall pattern that interval averages from zpool iostat smooth away.
• Correlating TXG sync duration with dirty data pressure on one dashboard separates “buffer too small” (disks idle, sync fast) from “backend saturated” (queues deep, sync slow) without manual kstat archaeology during an incident.
• Alerting on dirty data percentage and TXG stime together catches the pre-stall condition, which is when you can still fix it without user impact.
• Device-level disk metrics alongside ZFS-level latency make the misdiagnosis visible: the moment ZFS write latency spikes while device utilization stays flat, the throttle is your suspect.

Related guides

• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• ZFS ARC using all memory: the Linux default that eats your RAM
• ZFS zfs_arc_max: capping the ARC without starving read performance
• ZFS capacity planning: runway estimation before the pool fills