$ guides / zfs / zfs-write-latency-spikes ▌

Operations Guides

ZFS periodic write latency spikes: the TXG sync storm pattern

Your applications write fast for a few seconds, then every write stalls for 10 to 60 seconds, then everything is fast again. The stalls recur on a rough cycle, and between them the system looks completely healthy. Reads are fine. The pool is ONLINE. zpool status -x says all pools are healthy. Disk-level tools show the devices mostly idle, except for periodic bursts of intense write activity.

This is the TXG sync storm pattern, and the periodicity is the fingerprint. Write latency is bimodal: fast or blocked, with the blocked periods recurring roughly every zfs_txg_timeout seconds (default 5). It is distinct from steady-state slowness, which would indicate a simple throughput bottleneck. If your latency graph looks like a picket fence rather than a plateau, the write pipeline is backing up, not uniformly slow disks.

This pattern is diagnosable from two sources: the TXG history kstat and per-vdev latency. Operators routinely misdiagnose it as disk failure and replace hardware that is fine.

What this means

ZFS is batch-oriented on the write path. All writes accumulate in memory into transaction groups (TXGs), which are flushed to disk periodically, by default every 5 seconds via zfs_txg_timeout. Three TXGs are always in flight: one open (accepting new writes), one quiescing (finalizing), and one syncing (writing to disk).

A storm starts when one sync takes longer than the timeout. While the syncing TXG is still working, the open TXG keeps accepting writes, so it grows larger than intended. When the slow sync finally completes, that oversized open TXG becomes the next syncing TXG. It has more dirty data to flush, so it takes even longer. The next open TXG grows larger still. Each cycle feeds the next.

Meanwhile, ZFS protects itself from unbounded memory growth. Dirty data is capped at zfs_dirty_data_max (default 10% of RAM, capped at 4 GB on recent OpenZFS). When dirty data crosses zfs_delay_min_dirty_percent (default 60% of the max), ZFS starts inserting artificial delays into write syscalls. At the hard limit, writes block outright until the syncing TXG completes. Those delays and blocks are exactly the 10 to 60 second stalls your applications see.

flowchart TD
  A[Slow TXG sync starts] --> B[Open TXG keeps accepting writes]
  B --> C[Dirty data climbs toward zfs_dirty_data_max]
  C --> D[Write throttle engages at 60 percent]
  D --> E[Application writes stall for 10-60s]
  E --> F[Oversized TXG becomes next sync]
  F --> G[Next sync is bigger and slower]
  G --> B
  H[Root cause: burst, scrub, slow vdev, fragmentation] --> A

The cascade is self-reinforcing but not self-healing. It breaks only when the write rate drops long enough for a sync to complete inside the timeout, or when you remove whatever made the sync slow.

Common causes

Cause	What it looks like	First thing to check
Write burst exceeding flush capacity	Storm starts when rsync, bulk load, backup, or `zfs recv` begins; stops when it ends	`zpool iostat -v 1` for write throughput vs device capability
Scrub or resilver stealing I/O	Storm correlates exactly with the `scan:` line showing an active scrub or resilver	`zpool status` scan line
Single slow vdev	One device’s latency far above its peers; pool-wide sync waits for the slowest device	`zpool iostat -l -v 1` per-vdev latency
Fragmentation and high capacity	Sync times creep up over weeks; pool above ~80% with rising fragmentation	`zpool list -o name,cap,frag`
Device degradation	Slow vdev plus non-zero READ/WRITE/CKSUM counters, SATA resets in dmesg	`zpool status -v` and `dmesg`
atime updates on read-heavy workloads	Dirty metadata generated even without application writes	Check `atime` property on hot datasets

Two contributors are easy to miss. First, atime updates generate dirty metadata on read-only workloads, adding work to every sync; setting atime=off on hot datasets removes that. Second, some NVMe drives thermal-throttle under sustained writes, turning one side of a mirror into the slow vdev every sync must wait for. If your slow device is NVMe, check its temperature before assuming it is dying.

Quick checks

All of these are read-only and safe during an incident.

# 1. Confirm the pattern: recent TXG sync times and dirty bytes
# Header format varies slightly by OpenZFS version; verify the column header line first
head -2 /proc/spl/kstat/zfs/<pool>/txgs
tail -20 /proc/spl/kstat/zfs/<pool>/txgs

The txgs file lists recent transaction groups (history depth controlled by zfs_txg_history, default 100 entries) with columns including ndirty (dirty bytes) and stime (sync phase duration in nanoseconds). In a storm, you will see stime values well above 5 seconds and ndirty climbing across successive TXGs.

# 2. Check the configured timeout and dirty data limits
cat /sys/module/zfs/parameters/zfs_txg_timeout
cat /sys/module/zfs/parameters/zfs_dirty_data_max
cat /sys/module/zfs/parameters/zfs_delay_min_dirty_percent

# 3. Per-vdev latency: find the slow device
zpool iostat -l -v <pool> 1

# 4. Queue depth: is the backend saturated or is ZFS itself stuck?
zpool iostat -q -v <pool> 1

# 5. Is a scrub or resilver running?
zpool status <pool> | grep -A3 scan

# 6. Capacity and fragmentation context
zpool list -o name,size,alloc,free,cap,frag

# 7. Hardware-level evidence for a slow device
dmesg | grep -i -E "ata|sas|reset|timeout" | tail -30
zpool status -v <pool>

# 8. Latency histograms: confirm the bimodal distribution
zpool iostat -w <pool> 5

In zpool iostat -l output, watch the split between syncq_wait and asyncq_wait. Elevated asyncq_wait points at the TXG commit path (this pattern). Elevated syncq_wait points at the ZIL/SLOG path instead, a different problem with a different fix. Note that disk_wait values from zpool iostat -l are known to vary with the sampling interval (OpenZFS issue #7694), so treat them as relative indicators, not precise measurements.

How to diagnose it

Confirm periodicity. Watch zpool iostat <pool> 1 for a minute. If write throughput collapses and recovers on a cycle of roughly zfs_txg_timeout seconds, you have the storm. Steady low throughput is a different problem.
Read the TXG history. Pull stime and ndirty from /proc/spl/kstat/zfs/<pool>/txgs. The storm signature is stime consistently exceeding 2x the timeout (over 10 seconds by default) with ndirty growing across consecutive TXGs. Sync times in the 30 to 120 second range confirm a full cascade, and at that point applications are being throttled by the dirty-data delay mechanism.
Rule out the legitimate explanations first. Check zpool status for an active scrub or resilver, and check for a large in-progress snapshot deletion or zfs recv. These legitimately inflate sync times. If one is running and the storm coincides with it, you have your answer and the fix is scheduling or throttling, not hardware.
Localize to a vdev. zpool iostat -l -v 1 and zpool iostat -q -v 1. A TXG sync must wait for the slowest required I/O across all vdevs, so one device with elevated latency stalls the entire pool. If one vdev’s latency is 3x or more above its peers with queues building on that device, it is your bottleneck.
Classify the slow vdev. If the device also shows non-zero READ/WRITE/CKSUM counters in zpool status -v, or SATA/SAS resets in dmesg, it is failing. If counters are clean but latency is high under load, suspect thermal throttling (NVMe), a shared controller, or an SMR drive garbage-collecting under sustained writes.
If all vdevs look uniform, look up the stack. Check pool capacity and fragmentation (zpool list -o cap,frag). Above roughly 80% capacity, every sync does more allocator work as free space gets harder to find in large contiguous runs. Also check whether the workload itself is the burst: a bulk load, backup, or rsync that simply exceeds the pool’s sustained flush rate.
Distinguish from ZIL problems. If your applications are synchronous-write heavy (databases, NFS) and syncq_wait dominates while TXG stime looks normal, this is a ZIL/SLOG bottleneck, not a TXG storm. Do not tune TXG parameters for a ZIL problem.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
TXG `stime` (txgs kstat)	Direct measure of sync phase duration	Consistently > 2x `zfs_txg_timeout` (over 10s default)
TXG `ndirty`	Dirty bytes accumulated per TXG	Growing across successive TXGs
Dirty data vs `zfs_dirty_data_max`	How close the pipeline is to hard stall	Sustained above 80% of the limit
`zpool iostat -l` write `total_wait`	User-perceived write latency	Spikes above 10 seconds recurring periodically
`zpool iostat -w` write histogram	Tail latency the averages hide	p99 orders of magnitude above p50, bimodal buckets
`zpool iostat -q` pending per vdev	Saturation vs internal stall	Sustained pending much greater than active on one vdev
Per-vdev latency spread	Single slow device detection	One vdev p95 > 3x the median vdev
Pool capacity and fragmentation	Allocator overhead on the sync path	cap > 85% with frag rising
Scrub/resilver state	Legitimate sync inflation	Active scan during storm windows
Deadman events (`zpool events`)	A sync or I/O fully hung, not just slow	Any event; sync hung 10+ minutes

One boundary worth knowing: the deadman subsystem only fires when a TXG makes no progress for zfs_deadman_synctime_ms (default 10 minutes) or an individual I/O hangs past zfs_deadman_ziotime_ms (default 5 minutes). A storm with slow-but-progressing syncs will never trigger it, so absence of deadman events does not mean absence of this problem.

Fixes

Break the immediate storm

Pause the scrub or resilver if one is running. On supported OpenZFS versions, zpool scrub -p <pool> pauses the scan and returns I/O bandwidth to the write path. Resume it in a low-traffic window. The tradeoff: every paused scan extends the window in which corruption is unverified or redundancy is reduced.
Throttle the write source. If a bulk load, rsync, backup, or zfs recv triggered the storm, rate-limit it at the source. This is the fastest way to let syncs complete inside the timeout and drain the backlog.
Do not restart anything. A reboot or pool export/import does not fix a storm; the same workload against the same slow sync path will re-trigger it within minutes.

Fix a slow vdev

If the device is failing (error counters, resets in dmesg, SMART degradation): replace it proactively while redundancy is intact. Do not wait for it to fall off the bus.
If it is thermal throttling (clean counters, latency under load, high temperature): improve cooling or reduce sustained write rate. Verify with device temperature telemetry.
OpenZFS 2.4 note: 2.4.0 added the ability to temporarily sit out abnormally slow child vdevs on RAIDZ reads, reconstructing from parity. This does not remove the need to replace the slow device.

Fix the workload and scheduling

Move scrubs, resilvers where schedulable, backups, and bulk loads into windows that do not overlap with latency-sensitive write traffic. A scrub on a heavily loaded pool is the most common storm trigger.
Set atime=off on datasets where access-time tracking is not required, so read-heavy workloads stop generating dirty metadata for every sync.

Fix capacity and fragmentation

If the pool is above ~85% capacity with rising fragmentation, the sync path is doing expensive allocation work on every write. Prune snapshots, archive data, or expand the pool. There is no online defragmentation in ZFS; the only true fix for severe fragmentation is zfs send | zfs recv into a fresh pool, which is a migration project, not an incident response.

Tune the write pipeline (last resort, deliberately)

Raising zfs_dirty_data_max lets the pool absorb larger bursts before throttling, but it also makes each sync bigger and can lengthen sync times. It trades stall frequency for stall depth and RAM.
Raising zfs_txg_timeout reduces sync frequency but increases per-sync work and the amount of data at risk on a crash. Lowering it increases sync overhead.
These knobs change the shape of the problem, not its cause. Only touch them after you have identified why syncs are slow, and change them with measured before/after stime data.

Prevention

Baseline stime per pool. Know what your sync times look like in a quiet window. A storm is a deviation from that baseline, not an absolute number.
Trend capacity and fragmentation together. The cliff is the combination, not either metric alone. Act at 85% on write-heavy pools, not at 96%.
Schedule scrubs deliberately and alert when a scrub overlaps a latency incident, so the correlation is recorded rather than rediscovered each time.
Track per-vdev latency spread continuously. A device drifting to 2-3x its peers over weeks is your early warning for the most common storm trigger.
Monitor SMART and NVMe thermals alongside ZFS counters so a slow device is classified in minutes, not hours.
Keep storms off the critical path architecturally: if the pool serves latency-sensitive databases, do not co-schedule bulk ingest on the same pool.

How Netdata helps

Netdata collects ZFS pool health, capacity, fragmentation, and per-device error counters alongside system-level disk latency and throughput, so the pool view and the hardware view are on one timeline.
Per-second granularity catches the picket-fence latency signature that 1-minute averages smooth into a meaningless bump; the bimodal pattern is only visible at high resolution.
Correlating write latency spikes against scrub/resilver state, per-disk utilization, and throughput in one dashboard answers the first diagnostic question (is something stealing I/O?) without log archaeology.
Disk-level temperature and SMART metrics collected next to pool latency let you classify a slow vdev as thermal, failing, or saturated without switching tools.
Anomaly detection on write latency flags the onset of the periodic pattern before applications start timing out, which is the difference between pausing a scrub calmly and pausing it mid-incident.