ZFS ZIL commit stalls and errors: reading the intent-log kstats

A database that normally commits in single-digit milliseconds suddenly takes seconds per commit. NFS clients time out. zpool status shows the pool ONLINE, read latency looks fine, and zpool status -x says all pools are healthy. The breakage is confined to synchronous writes, which points at one subsystem: the ZFS Intent Log.

The ZIL guarantees synchronous write semantics. Every fsync, O_SYNC write, and NFS commit goes to the ZIL before the application gets its acknowledgment. When the ZIL backs up, because the SLOG is slow or failed, the pool cannot absorb log writes, or the write pipeline is throttled, every synchronous writer stalls at once. Reads from the ARC keep working, which is why this failure mode is confusing on first contact.

The primary instrumentation is a single kstat file: /proc/spl/kstat/zfs/zil. Two counters in it, zil_commit_stall_count and zil_commit_error_count, tell you whether the intent log itself is in trouble.

What this means

The kstat at /proc/spl/kstat/zfs/zil is a set of global, cumulative counters for ZIL activity across the whole system. The fields that matter operationally:

• zil_commit_count: total ZIL commits requested (one per fsync-class operation).
• zil_commit_writer_count: number of times the ZIL actually flushed to stable storage. This can be lower than zil_commit_count because ZFS batches commits.
• zil_commit_stall_count: commits that stalled waiting on the intent log.
• zil_commit_error_count: commits that failed with an error.
• zil_itx_count, zil_itx_copied_count, zil_itx_needcopy_count (and _bytes variants): intent transaction volume, useful for gauging sync write load.

Three constraints shape how you use this file:

1. The counters are global, not per-pool. On a multi-pool system you cannot tell from this file alone which pool is stalling. You correlate with per-pool latency and queue data to localize.
2. There is no zil_commit_latency metric. OpenZFS does not expose ZIL commit latency in kstats. Latency must be inferred from application-level fsync timing or from sync-write latency in zpool iostat -l (the syncq_wait columns).
3. The counters are cumulative. A single snapshot means little. What matters is whether the stall or error counters increment between samples.

Healthy state: a steady commit rate with zero stall and error increments. zil_commit_count should track your application’s sync write rate, and zil_commit_writer_count should track commits modulo batching.

Common causes

The fourth row is the most common misdiagnosis. When dirty data reaches the delay threshold (default 60% of zfs_dirty_data_max, via zfs_delay_min_dirty_percent), ZFS injects artificial delay into writers. To an application this looks identical to a ZIL stall: sudden commit hangs with a healthy-looking pool. Check TXG sync times and ZIL stall counts before blaming hardware. See ZFS dirty data throttling: the write delay that masquerades as slow disks for that pattern in depth.

Quick checks

All read-only and safe to run during an incident.

# 1. Snapshot the ZIL counters twice, 10 seconds apart, and diff
grep -E 'zil_commit_(count|writer_count|stall_count|error_count)' /proc/spl/kstat/zfs/zil > /tmp/zil.a
sleep 10
grep -E 'zil_commit_(count|writer_count|stall_count|error_count)' /proc/spl/kstat/zfs/zil > /tmp/zil.b
diff /tmp/zil.a /tmp/zil.b

# 2. Check SLOG device state (the logs section of the vdev tree)
zpool status -v

# 3. Sync vs async queue wait, per pool - high syncq_wait points at ZIL/SLOG
zpool iostat -l 1 5

# 4. Queue depths - high sync queue with low async queue points at ZIL/SLOG
zpool iostat -q -v 1 5

# 5. Recent TXG sync durations (stime column) vs the 5s default timeout
tail -10 /proc/spl/kstat/zfs/<pool>/txgs

# 6. Is a scrub or resilver running right now?
zpool status | grep -A3 'scan:'

# 7. Hardware-level errors correlating with the stalls
dmesg | grep -i -E 'ata|sas|reset|timeout' | tail -20

Interpreting check 1: if zil_commit_count increments but stalls and errors stay flat, the ZIL is keeping up; your problem is elsewhere (likely application-side or TXG sync). If zil_commit_stall_count increments in step with commits, the intent log is the bottleneck. Any increment in zil_commit_error_count is abnormal and warrants hardware investigation.

How to diagnose it

The stall counter tells you the ZIL is waiting, but not what it is waiting on. Work outward from the log to the pool to the write pipeline.

flowchart TD
  A[zil_commit_stall_count incrementing] --> B{SLOG present?}
  B -->|yes| C{SLOG healthy in zpool status?}
  C -->|no| D[Failed SLOG: ZIL fell back to pool vdevs]
  C -->|yes| E[SLOG saturated or slow: check device latency and SMART wear]
  B -->|no| F[ZIL on pool vdevs]
  F --> G{One vdev slow in zpool iostat -v?}
  G -->|yes| H[Dying or saturated device: check dmesg and SMART]
  G -->|no| I{TXG stime well over 5s, dirty data near max?}
  I -->|yes| J[Dirty-data throttle / TXG sync hang, not a ZIL fault]
  I -->|no| K[Scrub or resilver contending for I/O]

1. Confirm the stall is real and ongoing. Take two samples of /proc/spl/kstat/zfs/zil 10-30 seconds apart. Only the delta matters. A historical bump from a past event is not your incident.

2. Quantify commit pressure. The rate of zil_commit_count between samples is your sync write arrival rate. Compare zil_commit_writer_count to zil_commit_count: writer count well below commit count means batching is absorbing commits, which is normal; writer count matching commit count 1:1 under load means commits are not being batched.

3. Check for errors. Any increment in zil_commit_error_count shifts the investigation from performance to hardware. Go straight to zpool status -v error columns and dmesg.

4. Localize to a pool. The kstat is global, so use per-pool evidence: zpool iostat -l 1 and look for the pool whose write syncq_wait spiked. High sync queue wait with normal async wait is the signature of a ZIL/SLOG problem.

5. Interrogate the SLOG if one exists. In zpool status -v, find the logs vdev. A FAULTED log device drops the pool to DEGRADED and ZFS falls back to writing the ZIL on the main pool, with a corresponding latency jump. The failure is easy to miss because the log vdev sits in its own section of the vdev tree and the pool keeps serving I/O. If the SLOG is ONLINE, check its device-level latency and SMART wear indicators. There is no ZFS-level kstat for SLOG write volume; l2_write_bytes in arcstats is L2ARC writes, not SLOG.

6. Rule out the dirty-data throttle. Check the pool’s TXG history: tail /proc/spl/kstat/zfs/<pool>/txgs and read the stime column (sync duration). If stime is consistently 2x or more over zfs_txg_timeout (default 5s) and dirty data is high, the stall you are chasing is the write pipeline backing up, not the ZIL itself. The ZIL stall counter is a symptom here.

7. Rule out background I/O. Check the scan: line for an active scrub or resilver. Both legitimately inflate latency and can push a marginal SLOG or pool over the edge.

8. Correlate with the application. Since there is no ZIL latency metric, confirm impact at the source: database commit latency, NFS server response time, or application fsync timing. This also gives you the before/after measurement for any fix.

Metrics and signals to monitor

Fixes

Failed or failing SLOG

Replace the device: zpool replace <pool> <old-slog> <new-slog>, or remove the failed log vdev and add a new one. These are mutating pool operations, so confirm the vdev names against zpool status before running them; a replace against the wrong device can destroy redundancy on the wrong vdev. An unmirrored SLOG failing during a crash can lose recently synced transactions, so production SLOGs should be mirrored. If the SLOG failed from wear, verify the replacement has the endurance profile for concentrated sync writes and put SMART wear monitoring on it. There is no ZFS-level wear metric.

No SLOG on a sync-heavy pool

If the ZIL lives on pool vdevs (spinning disks especially) and the workload is databases or NFS, adding a fast, power-loss-protected SLOG device is the structural fix. NFS on ZFS forces synchronous writes, so this combination without a SLOG is a standing incident waiting for load. Tradeoff: a SLOG only helps synchronous writes; async write throughput is unaffected, and a slow or undersized SLOG can be worse than a pool-resident ZIL.

Single slow pool device

If one vdev shows much higher latency than its peers and dmesg shows resets or timeouts, you are in the dying-disk pattern. Check SMART, and if the pool has redundancy, consider offlining the device proactively before it faults mid-incident. Do not offline a device from a non-redundant or already-degraded vdev; that converts a performance problem into data unavailability.

Dirty-data throttle, not ZIL

If TXG stime is far over the timeout and dirty data sits above the 60% delay threshold, the ZIL counters are collateral. The fixes live on the write-pipeline side: find the slow vdev, pause an aggressive scrub (zpool scrub -p <pool> on supported versions), or throttle the write surge at the source. Do not start by raising zfs_dirty_data_max: it permits bigger bursts but also bigger TXG syncs and higher RAM use, and it does nothing if the disks genuinely cannot keep up. The full decision tree is in ZFS dirty data throttling: the write delay that masquerades as slow disks.

Commit errors

zil_commit_error_count incrementing is hardware until proven otherwise. Pull zpool status -v error columns for the log vdev and data vdevs, check dmesg for link resets, and check SMART. Checksum errors on multiple unrelated devices at once point at RAM or the controller, not the disks. See ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks.

One thing to never do as a fix: setting sync=disabled on a dataset to make the stalls disappear. It bypasses the ZIL and converts sync writes to async, which sacrifices durability of the most recent transactions on crash. For databases and NFS this trades a performance incident for a future data-loss incident.

Prevention

• Trend the counters, do not spot-check. zil_commit_stall_count and zil_commit_error_count are cumulative; they are only meaningful as rates over time. Export them to a time-series system so a stall storm has history attached.
• Alert on increment, not on value. Any sustained increment in the stall counter is ticket-level; any increment in the error counter is a hardware investigation.
• Monitor the SLOG as a first-class device. The pool keeps running when the SLOG dies, so it is easy to miss. Track log vdev state, log vdev error counts, and device-level SMART wear explicitly.
• Baseline sync write latency. Since there is no ZIL latency kstat, keep a baseline of application fsync latency and syncq_wait so a 10x regression is obvious rather than discovered by users.
• Size for the sync workload before it arrives. NFS and database datasets on pool-resident ZIL over HDDs is a known-bad layout. Add the SLOG at build time, mirror it, and monitor its wear.

How Netdata helps

• Netdata collects the ZFS kstats, including the ZIL counters, as per-second time series, so zil_commit_stall_count and zil_commit_error_count become rates you can alert on instead of numbers you have to diff by hand.
• Correlating the ZIL stall rate with per-pool write latency and queue depth on one dashboard is exactly the localization step (which pool, sync path or async path) that the global kstat cannot do alone.
• TXG sync duration alongside dirty-data pressure lets you separate a genuine ZIL bottleneck from the dirty-data throttle masquerading as one, without bouncing between /proc files mid-incident.
• Device-level disk latency and SMART wear next to SLOG vdev state surfaces a dying log device before sync latency does.
• Historical retention means the post-incident question “when did the stalls actually start” has an answer, which matters for tying the onset to a scrub schedule, a deploy, or a workload change.

Related guides

• ZFS dirty data throttling: the write delay that masquerades as slow disks
• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption