$ guides / zfs / zfs-txg-sync-time-high ▌

Operations Guides

ZFS TXG sync time high: the most diagnostic write-path signal operators ignore

Applications are stalling on writes. zpool status -x says all pools are healthy. Disk latency looks mostly fine, or at least inconsistent with the severity of the application impact. The signal that explains what is actually happening sits in one file most operators never open: /proc/spl/kstat/zfs/<pool>/txgs, specifically the stime field, which records how long each transaction group took to commit to stable storage.

TXG sync time sits between “the application sees slow writes” and “a disk is slow” and explains the relationship between them. When sync time climbs, dirty data accumulates in the open TXG, ZFS starts throttling writers, and applications experience the periodic write freezes that are so often misattributed to hardware failure or application bugs.

What this means

ZFS batches writes into transaction groups (TXGs) and flushes them to disk on a cadence controlled by zfs_txg_timeout (default 5 seconds). There may be up to one TXG in each active state—open (accepting writes), quiescing (finalizing), and syncing (writing to disk)—and there is always an open TXG. The stime field is the duration of the sync phase, in nanoseconds.

The failure mechanism is a feedback loop:

flowchart TD
  A[Syncing TXG takes too long] --> B[Open TXG accumulates dirty data]
  B --> C[Next TXG is even larger]
  C --> D[Next sync takes even longer]
  D --> A
  B --> E[Dirty data crosses zfs_delay_min_dirty_percent - 60% of max]
  E --> F[ZFS throttles writers]
  F --> G[Application write latency spikes to seconds]

The throttle is the intended memory-protection behavior, so the system is working as designed when it stalls your writes. The disks are simply not draining dirty data fast enough. Severity tiers for stime:

Normal: under 2 seconds
Elevated: over 5 seconds (the default zfs_txg_timeout)
Serious: over 15 seconds, write latency spikes are occurring
Critical: over 30 seconds, applications are actively throttled

Two properties of this signal matter for diagnosis. First, TXG sync time is per-pool, not per-dataset. One busy dataset degrades write latency for every dataset in the pool, so do not assume the loudest application is the cause. Second, stime alone is not page-worthy: scrubs, resilvers, large snapshot deletions, zfs recv, and pool import all legitimately extend sync times. The composite TXG sync hang pattern is stime sustained above 3x the timeout for more than 60 seconds, dirty data approaching zfs_dirty_data_max, write latency spiking, and no scrub, resilver, or import in progress to explain it.

Common causes

Cause	What it looks like	First thing to check
Slow or degraded device	One vdev much slower than peers; stime elevated across the pool	`zpool iostat -v 1`, then `dmesg` for link resets
Fragmentation	FRAG above 50%, stime creeping up over weeks, worst on HDDs	`zpool list -o name,frag`
Capacity allocator overhead	Pool above 85%, stime rising with capacity, writes and deletes both slow	`zpool list` CAP column
Dedup writes	DDT not fitting in RAM, every write pays a DDT lookup, read performance also collapsed	`zpool get dedup`, `zpool status -D`
CPU-bound compression	stime high while device latency is low; `z_compress` threads saturating cores	`top -H` for `z_compress`, `z_cksum` threads
Legitimate background work	stime spikes only during scrub, resilver, or large snapshot destroy windows	`zpool status` scan line

Quick checks

All read-only.

# Recent TXG sync times (replace tank with your pool)
cat /proc/spl/kstat/zfs/tank/txgs | tail -20

# Confirm the timeout and history tunables
cat /sys/module/zfs/parameters/zfs_txg_timeout
cat /sys/module/zfs/parameters/zfs_txg_history

# Dirty data pressure vs the limit
cat /sys/module/zfs/parameters/zfs_dirty_data_max
cat /sys/module/zfs/parameters/zfs_delay_min_dirty_percent

# Per-vdev throughput - find the slow device
zpool iostat -v 1

# Per-vdev latency breakdown
zpool iostat -l 1

# Fragmentation and capacity
zpool list -o name,size,alloc,free,cap,frag

# Is a scrub or resilver running right now
zpool status | grep -A3 "scan:"

# Link resets, timeouts, controller errors
dmesg | grep -i -E "ata|sas|reset|timeout"

Notes on reading the txgs file:

The header line is: txg birth state ndirty nread nwritten reads writes otime qtime wtime stime. All time fields are nanoseconds. Column positions vary slightly by OpenZFS version, so verify the header before scripting against it.
stime is the sync phase duration, the field you want. ndirty shows how much dirty data the TXG carried; dividing nwritten by stime gives effective flush throughput for that TXG.
The file holds zfs_txg_history entries. OpenZFS 0.8.0 and later default to 100; OpenZFS 0.6/0.7 defaulted to 0, so the file may remain empty until the value is raised. At a 5-second cadence, 100 entries is roughly the last 8 minutes.

How to diagnose it

Quantify stime. Pull the last 20 rows of the txgs file and convert stime to seconds. Establish whether you are in the elevated (5-15s), serious (15-30s), or throttled (30s+) band, and whether it is sustained or spiking around a background operation.
Rule out legitimate causes. Check the scan: line in zpool status. A running scrub or resilver competes for I/O bandwidth and inflates sync time by design. Large snapshot deletions and zfs recv do the same. If stime returns to normal when the background operation finishes, you do not have an incident; you have a scheduling problem.
Split device latency from ZFS-internal latency. Run zpool iostat -l 1 and compare total_wait (queue plus disk) against disk_wait (disk only). If disk_wait is high, the backend device is the bottleneck. If total_wait is high but disk_wait is low, the delay is inside ZFS: allocator work, queue contention, or CPU-bound processing. This single comparison routes the rest of the investigation.
If devices look slow, find the outlier. zpool iostat -v 1 shows per-vdev numbers. In a mirror or RAIDZ group, the slowest device gates the whole group. Look for one device with latency several times its peers, then check dmesg for SATA/SAS resets or timeouts and check zpool status -v for growing READ, WRITE, or CKSUM counters on that device. SATA link reset storms, SMR drives garbage collecting, and controller saturation are the usual culprits.
If devices look fine, check fragmentation and capacity. zpool list -o name,cap,frag. Above roughly 85% capacity, metaslab allocation gets expensive and every write carries allocator overhead. Fragmentation above 50% on a write-heavy pool scatters allocations and turns sequential writes into random I/O. The two compound each other; a pool at 88% capacity with 55% fragmentation is already in the degradation zone even with healthy disks.
Check dedup and compression if enabled. If dedup=on, check whether the DDT fits in RAM (zpool status -D). A DDT that spills to disk makes every write pay a random read first, which destroys sync time. For compression, check whether z_compress or z_cksum kernel threads are saturating cores in top -H; ZSTD at high levels on fast storage can make sync CPU-bound while the disks sit idle.
Measure dirty data pressure. Compare observed dirty data (the ndirty column) against zfs_dirty_data_max. The write throttle engages at zfs_delay_min_dirty_percent (default 60% of max). If TXGs consistently carry dirty data near that threshold, writers are being delayed by design and the stall will not resolve until flush capacity catches up.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
TXG `stime`	Direct measure of flush capacity vs write load	Sustained above 2x `zfs_txg_timeout` (>10s default), or trending upward over hours
TXG `ndirty`	How much data each sync must drain	Approaching the 60% throttle threshold of `zfs_dirty_data_max`
`total_wait` vs `disk_wait`	Routes diagnosis: device problem vs ZFS-internal problem	Large gap between the two
Per-vdev latency (`zpool iostat -v -l`)	Finds the single slow device gating a vdev group	One device 3x+ slower than peers
Pool capacity and fragmentation	Allocator overhead grows non-linearly near full	CAP above 85% with FRAG rising
ZFS thread CPU	Compression and checksum work can bound sync time	`z_compress`/`z_cksum` saturating cores

Fixes

Slow or degraded device

If one device is consistently the outlier and the pool has redundancy, offlining or replacing it is the direct fix. Check SMART data and error counters first to confirm the device is actually failing rather than suffering a controller or cabling issue shared with peers. Do not offline a device in a vdev that has no remaining redundancy; that converts a performance problem into an availability problem.

Fragmentation and capacity pressure

There is no in-place defragmentation for ZFS. Scrub does not defragment. The only real fix for severe fragmentation is zfs send | zfs recv into a fresh pool. For capacity pressure, prune snapshots (zfs list -t snapshot -o name,used -s used to find the worst offenders), archive data, or expand the pool. Treat 85% as the action threshold on write-heavy pools, not 95%.

Dedup

If the DDT no longer fits in RAM, there is no cheap fix. Setting dedup=off affects only new writes; existing deduplicated data keeps its DDT overhead. The durable fix is migrating data to a non-dedup dataset via send/recv. Plan for this to be slow.

CPU-bound compression

If compression threads are the bottleneck, switch hot datasets to lz4 (cheap) or a lower ZSTD level. This affects new writes only, so relief is gradual.

Throttle tuning as a pressure valve

Raising zfs_txg_timeout or zfs_dirty_data_max changes the shape of the problem, not the capacity underneath it. A longer timeout makes each sync bigger and slower; a bigger dirty data limit delays the throttle at the cost of more RAM and larger syncs. These are reasonable short-term levers for bursty workloads on undersized buffers, but if disk_wait says the devices cannot drain the data, no tunable fixes that.

Prevention

Trend stime, do not sample it. The txgs file holds only the last 100 TXGs. Export stime to a time-series system so you can see the slow upward drift that precedes the cliff, and so incident review has history to work with.
Alert on the composite, not the raw value. Page when stime exceeds 3x the timeout for more than 60 seconds with no scrub, resilver, or import in progress. A raw stime alert will false-fire during every scrub window.
Track capacity and fragmentation together. Capacity rising plus fragmentation rising is the leading indicator for allocator-driven sync slowdown. Plan expansion at 75-80%, act at 85%.
Baseline your pools. Know normal stime, normal per-vdev latency, and normal flush throughput per pool. Deviations from baseline are the signal; absolute numbers vary by media and topology.
Schedule scrubs and resilvers deliberately. They legitimately inflate sync time. If your write-heavy window overlaps your scrub window, you will chase phantom incidents.

How Netdata helps

Netdata collects the txgs kstat per pool, so stime and ndirty become continuous time series instead of an 8-minute rolling window you had to catch by hand.
Per-vdev latency and throughput sit next to TXG sync time on the same dashboard, so the “is it a device or is it ZFS-internal” split from step 3 is a visual correlation, not two terminal sessions.
Pool capacity and fragmentation are charted alongside write-path metrics, which makes the slow capacity-fragmentation creep visible weeks before stime crosses into the serious band.
Dirty data relative to its limits is surfaced directly, so you can see the throttle threshold approaching before applications start stalling.
ZFS device error counters and pool state are monitored continuously, so a slow device dragging sync time shows up with its READ/WRITE/CKSUM history attached.