ZFS SLOG device failed: sync latency 100x worse while the pool stays ONLINE

NFS clients are timing out. Database commit latency has jumped from microseconds to tens or hundreds of milliseconds. zpool status -x says “all pools are healthy.” Reads are fine, capacity is fine, scrubs are clean.

This is the SLOG failure trap. A SLOG (separate intent log) failure does not degrade pool redundancy, so the pool state stays ONLINE. But the ZIL has silently fallen back to writing on the main pool vdevs, and every synchronous write now pays the full latency of your data disks instead of the fast log device. For sync-heavy workloads, that is a 10x to 100x latency regression with zero change in pool health state.

A SLOG failure by itself does not lose data. But nothing in the standard “is the pool healthy” check will tell you it happened.

What this means

Every synchronous write (O_SYNC, fsync, NFS commits, database WAL flushes) must be acknowledged through the ZIL before ZFS tells the application the write is durable. With a healthy SLOG, those commits land on a dedicated fast device and return in sub-millisecond to single-digit-millisecond time. ZIL data on the SLOG is write-only during normal operation; it is only ever read during pool import after a crash.

When the SLOG device fails, ZFS does not stop. It starts writing the ZIL to the main pool vdevs, where it competes with normal data I/O at whatever latency your spinning disks or RAIDZ can deliver. Reads keep coming from the ARC at full speed, async writes continue through the normal TXG path, and the pool state stays ONLINE because log devices carry no data redundancy responsibility.

flowchart TD
  A[Application fsync / O_SYNC / NFS commit] --> B[ZIL commit]
  B -->|SLOG healthy| C[Fast SLOG device - sub-ms ack]
  B -->|SLOG failed - silent fallback| D[ZIL on main pool vdevs]
  D --> E[Sync latency 10x-100x worse]
  C --> F[Pool state: ONLINE]
  D --> F
  F --> G["zpool status -x: all pools are healthy"]

Data loss requires a compound failure: the SLOG fails AND the host crashes or loses power before the next TXG commit flushes dirty data to the main pool. In that window, recently acknowledged sync writes that lived only in the failed SLOG are gone. A SLOG failure alone, with the system still running, loses nothing.

Common causes

That last row matters. Before blaming hardware, rule out TXG sync pressure and ZIL saturation. A failed SLOG has a specific signature: sync write latency collapses while reads and pool state stay normal, and the logs section of zpool status shows the dead device.

Quick checks

All read-only and safe to run during an incident.

# 1. The check that actually catches this: full status including the logs section
zpool status -v <pool>

# 2. Confirm what pool-level health checks are telling you (likely: nothing)
zpool status -x

# 3. ZIL placement: watch whether commits are still landing on the SLOG
cat /proc/spl/kstat/zfs/zil

# 4. Write latency with queue breakdown; watch syncq_wait on writes
zpool iostat -l <pool> 1

# 5. Queue depths; a deep sync write queue with flat async queues points at the ZIL path
zpool iostat -q -v <pool> 1

# 6. Kernel view of the device failure
dmesg | grep -i -E "ata|sas|nvme|reset|timeout"

# 7. Wear state of the SLOG device itself (use the actual device name)
smartctl -a /dev/<slog-device>

Two things on checks 1 and 2. The logs section sits below the data vdevs in zpool status output, and it is exactly what gets missed when operators skim the top of the tree or only run zpool status -x. And on OpenZFS 2.2.x there are reports (openzfs/zfs#16245) of a faulted log device not transitioning the pool to DEGRADED; the pool state reads ONLINE with FAULTED devices listed underneath. zpool reopen can force the state to update. Do not assume DEGRADED will ever appear.

How to diagnose it

1. Confirm the application symptom is sync writes. Database commit latency, NFS write latency, fsync-heavy workloads degraded; bulk reads and async writes normal. If everything is slow including reads from disk, this is a different problem (check TXG sync times and vdev latency instead).

2. Read the full zpool status -v. Look at the logs section explicitly. A SLOG showing FAULTED, UNAVAIL, or REMOVED, or with climbing WRITE errors, confirms the failure. Do not stop at the pool-level state line.

3. Confirm the fallback in the latency data. zpool iostat -l 1 should show write total_wait and syncq_wait far above baseline while read latency stays flat. High syncq_wait with normal asyncq_wait is the ZIL path signature.

4. Check where ZIL blocks are landing. /proc/spl/kstat/zfs/zil tracks ZIL writes by placement: zil_itx_metaslab_slog_bytes for commits on the SLOG versus zil_itx_metaslab_normal_bytes for commits on the main pool. Under sync load, a healthy pool shows the slog bytes counter climbing. If slog bytes are flat while normal bytes climb, the ZIL has fallen off the SLOG. These counters are global, not per-pool, so on multi-pool systems interpret with care.

5. Rule out the impostors. Check /proc/spl/kstat/zfs/<pool>/txgs (stime field): if TXG sync times are also blowing past zfs_txg_timeout, you may have a broader write-path problem (slow data vdev, fragmentation, saturation) rather than a clean SLOG failure. Check whether a scrub or resilver is running.

6. Identify the failed hardware. Map the SLOG device to a physical device and serial, pull SMART data, and check dmesg for the failure mode. If the device shows worn-out NAND, plan for a higher-endurance replacement, not a like-for-like consumer part.

Metrics and signals to monitor

Two instrumentation gaps to design around. ZIL latency is not exposed anywhere in kstats, so it must be inferred from application-level fsync timing and the syncq_wait columns. And there is no ZFS-level counter for SLOG write volume; l2_write_bytes in arcstats is L2ARC, not SLOG. SLOG endurance monitoring has to come from device-level SMART data.

Fixes

Replace the failed SLOG

Physically install the replacement device, then:

# If the old device is still present in the pool topology
zpool replace <pool> <old-slog-device> <new-slog-device>

# If the old device is fully gone, remove it then add the new one
zpool remove <pool> <old-slog-device>
zpool add <pool> log <new-slog-device>

zpool remove on a log device is orderly and safe: the pool keeps running with the ZIL on the main vdevs throughout, and no pool data is touched. A failed SLOG is the messy version of the same thing.

One caution: on some configurations, notably certain encrypted pools, log device removal or replacement can trip over ashift mismatches between old and new devices. If zpool remove or zpool replace errors out on sector size, match the original device’s ashift explicitly with -o ashift=N on the replace.

Mirror the SLOG this time

An unmirrored SLOG is a single point of failure for exactly the scenario you are in. A mirrored SLOG (zpool add <pool> log mirror <dev1> <dev2>) survives a single device loss without the latency collapse, and it narrows the crash-during-failure data loss window. The capacity cost is trivial: the SLOG only needs to hold a few seconds of sync write throughput, so small devices (16 GB class, over-provisioned) are sufficient.

Mirror with care on device selection. Two consumer SSDs without power-loss protection in the same host do not protect you against the power-event scenario: both sides lose their volatile write caches simultaneously. Use devices with power-loss protection for a SLOG, mirrored or not.

Do not “fix” it with sync=disabled

Setting sync=disabled on a dataset bypasses the ZIL and makes the latency problem vanish, at the cost of losing acknowledged writes on a crash. This is how operators turn a performance incident into a silent corruption incident. It is never acceptable for databases or NFS exports, and it does not fix the failed hardware anyway.

Prevention

• Monitor the logs section, not just pool state. Alert on any log or cache vdev that is not ONLINE, and on any non-zero error counter in that section. Pool-level health checks structurally cannot see this failure.
• Monitor sync write latency directly. Track write total_wait/syncq_wait or application commit latency against a baseline. A 5-10x step change with flat reads is the alert that catches this even when device state monitoring misses it.
• Track SLOG wear as a capacity resource. SLOG SSDs take the pool’s entire sync write stream and fail with no graceful degradation. Trend SMART wear and replace at 70-80% of rated endurance on production systems.
• Mirror the SLOG from day one on any pool serving sync-critical workloads (NFS, databases). The cost is one extra small SSD.
• Use power-loss-protected devices. A SLOG that acknowledges writes it has not persisted reopens the data loss window on every power event.

How Netdata helps

• Pool and vdev state visibility: Netdata’s ZFS pool collector surfaces device health, so a FAULTED SLOG shows up as its own signal rather than being averaged away into an ONLINE pool state.
• SMART correlation: Device wear and error attributes sit next to the ZFS signals on the same dashboard, so “wear hit 95%, then sync latency stepped up” is one screen, not two tools.
• Disk-level latency split: Per-disk latency charts make the shift visible: the SLOG device goes quiet while data vdev write latency jumps.
• The correlation that matters: SLOG state change, sync write latency step change, flat read latency, and flat pool state, all on one timeline. That combination is a near-unambiguous SLOG failure fingerprint and cuts the diagnosis from an afternoon of zpool status archaeology to minutes.

Related guides

• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS monitoring checklist: the signals every production pool needs
• How ZFS actually works in production: a mental model for operators
• ZFS monitoring maturity model: from survival to expert
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption