ZFS pool history: the forensic log almost nobody watches

Every ZFS pool keeps a built-in audit log. Every property change, every snapshot creation and destruction, every import, export, key operation, and delegation change is recorded by zpool history, with timestamps, and with the user and hostname if you ask for them. Almost nobody monitors it.

This is a problem in two directions. Forensically, when something goes wrong (a dataset destroyed at 3 a.m., sync suddenly disabled on a database dataset, a pool exported that nobody admits to exporting) the answers were in the history all along, often already rotated out by the time anyone looks. On the security side, mass snapshot destruction is exactly what ransomware or an attacker covering tracks looks like, and an unauthorised sync=disabled or delegation change is a privilege escalation signal. Both are visible in one place, and that place is not wired into anything by default.

This article covers what zpool history actually is mechanically, what it does and does not record, the ring-buffer limits that silently eat your evidence, and how to turn it into a monitored, off-box audit trail.

What it is and why it matters

zpool history is not a log file. It is a DMU object stored in the pool’s metadata (the MOS), written transactionally as part of the same copy-on-write model as everything else in ZFS. ZFS automatically logs every successful zfs and zpool command that modifies pool state. That includes:

• Property changes (zfs set, zpool set)
• Snapshot and dataset creation, destruction, and rollback
• Pool imports and exports
• Encryption key load, unload, and change operations
• Delegation changes (zfs allow / zfs unallow)
• Surprisingly, zfs send, which is read-only but still recorded

Because the history is pool metadata, it is checksummed and protected like any other ZFS metadata. An attacker who wants to erase it needs pool-level write access, at which point you have bigger problems. But there is a subtler integrity issue: the buffer is finite, it rotates, and on busy pools it rotates fast. The tampering you should worry about is not deletion. It is your own snapshot automation flushing the evidence out of the buffer before an incident.

How it works: the ring buffer

The history is implemented as a ring buffer whose size is computed once, at pool creation:

• 0.1% of pool space at creation time
• Capped at 1 GB
• Floored at 128 KB

Three consequences follow, and all three bite operators in practice:

1. The size never recalculates. If you create a small pool and later expand it (vdev additions, larger replacement disks), the history buffer stays at its original size. A pool that grew from 2 TB to 80 TB still has the buffer it was born with.
2. Old records are overwritten silently. The history object tracks how many records were lost to rotation, but nothing alerts on it. Evidence disappears with no event, no error, and no log line.
3. The zpool create record is never overwritten. The buffer preserves the creation record permanently; everything after it is fair game for eviction.

The viewing flags matter for forensics:

# Default view: commands and timestamps
zpool history tank

# Long format: adds user name, hostname, and zone
zpool history -l tank

# Internal events: adds internally logged ZFS events
# (txg changes, feature enables, pool opens/imports)
zpool history -i tank

The -l flag is the one that turns history from a list of commands into an audit trail: it answers “who” and “from where”, not just “what”. The -i flag reveals internally logged events that the default view does not show; there are reports of pools where the plain view appears to show only recent import/export cycles while -i still shows the full history back to creation. If the default view looks suspiciously short, check -i before concluding records were lost.

flowchart LR
  A[Admin runs zfs or zpool command] --> B[History ring buffer in pool MOS]
  B --> C[history_event generated]
  C --> D[ZED event daemon]
  D -->|default zed.rc: subclass excluded| E[Nothing reaches syslog]
  D -->|exclusion removed| F[syslog or journal]
  F --> G[Off-box log store]
  B -.->|buffer full: oldest records overwritten| H[Evidence lost silently]

The snapshot noise problem

This is the gotcha that makes naive reliance on zpool history fail exactly when you need it.

Automated snapshot tooling (sanoid, zfs-auto-snapshot, cron jobs, replication systems) creates and destroys snapshots constantly, and every one of those operations is a history record. In one documented case on a large pool, 99.8% of roughly 670,000 history entries were snapshot create/destroy operations; barely 1,400 entries were anything else. On a pool with a small creation-time buffer and aggressive snapshot scheduling, meaningful events (property changes, exports, key operations, vdev changes) are pushed out of the ring buffer by pure churn.

There is an open OpenZFS feature request (issue #13374, open since 2022) for a separate low-churn event buffer so meaningful events survive snapshot noise. As of now there is no merged fix. There is also no zpool history --clear or pruning command of any kind: the history is an immutable transactional part of pool metadata. The only way to get a clean history is to create a new pool and migrate via send/receive.

Practical implications:

• Do not treat on-box history as your retention mechanism. Treat it as a convenience cache of recent events. Your real audit trail has to live off-box.
• Ship events as they happen, not by polling. By the time a daily zpool history scrape runs, the record you needed may already be rotated out on a churny pool.
• Filter for signal, not volume. The events that matter for change control and security are a tiny minority: set, destroy, rollback, export, import, key, allow, unallow, offline, online, replace, clear.

The ZED default that hides everything

ZED (the ZFS Event Daemon) can forward pool events to syslog, and history records generate a history_event subclass. But the shipped default in /etc/zfs/zed.d/zed.rc is:

# Default: history events are excluded from syslog
ZED_SYSLOG_SUBCLASS_EXCLUDE="history_event"

That single line is why most operators have never seen a pool administrative action in their logs. The events are generated, ZED sees them, and ZED deliberately drops them before they reach syslog. If your audit trail strategy is “ZFS stuff goes to the log server like everything else”, it does not, by default.

To fix it, edit zed.rc and remove history_event from the exclusion (or comment the line out), then restart ZED. From that point, every recorded administrative action flows into syslog, where your existing log shipping (rsyslog, journald forwarding, an agent) can carry it off-box. Restarting ZED is safe on a live pool; it does not touch pool state, only event handling. Be aware of the volume tradeoff on snapshot-heavy pools: filter snapshot create/destroy at the syslog or pipeline layer rather than at ZED, so the off-box store keeps the meaningful minority without paying for the churn.

What to look for: the forensic queries

These are the searches worth being able to run cold, at 3 a.m., without thinking. All are read-only.

# Snapshot and dataset lifecycle: who destroyed what, and when
zpool history -l tank | grep -E "snapshot|destroy|rollback" | tail -20

# Property changes: sync=disabled, compression, mountpoint, shares, readonly
zpool history -l tank | grep "set " | tail -20

# Import and export activity
zpool history -l tank | grep -E "export|import" | tail -10

# Encryption key operations
zpool history -l tank | grep -i key | tail -10

# Delegation changes (privilege escalation vector)
zfs allow tank
zpool history -l tank | grep "allow\|unallow"

What each pattern catches:

• Mass snapshot destruction. Some bulk deletions are legitimate (retention pruning, replication cleanup), which is why this is a ticket-level signal, not a page. But a burst of destroys outside the scheduled window, especially from an unexpected user in the -l output, is what ransomware covering its tracks looks like.
• sync=disabled appearing. Removing synchronous write semantics is a performance “optimisation” that sacrifices crash consistency. It is also what an attacker optimising for exfiltration speed might do, and what a well-meaning engineer does without telling anyone. Either way you want to know immediately. Never accept sync=disabled on database datasets as a standing rule.
• Unexpected exports. An export outside maintenance means either an operational accident or someone preparing to import the pool elsewhere. In multihost or shared-storage environments, unexpected imports must be verified against MMP expectations.
• Key operations and delegation changes. A key load making an encrypted dataset accessible, or a zfs allow granting destroy/send/mount permissions to a new principal, are both privilege-boundary events. They should map one-to-one to change tickets. Any that do not are an incident.

Where this fits in a monitoring practice

Teams usually get capacity, scrub, and ARC monitoring in place and never reach the audit layer. The cost of adding it is low, because the mechanism already exists. You are not instrumenting anything new; you are removing one exclusion line and shipping a stream you already generate.

A workable setup:

1. Remove the ZED exclusion for history_event in /etc/zfs/zed.d/zed.rc and restart ZED.
2. Forward syslog or journal off-box with whatever log pipeline you already run. On-box history is a cache; the log store is the record.
3. Alert on the meaningful minority. Ticket on: any set on security-relevant properties, destroys outside the snapshot-pruning window, any export, any key operation, any allow/unallow. These are ticket-severity rather than page-severity in a generic setup, because legitimate automation does some of them routinely; page only when you have change-control integration to compare against.
4. Baseline snapshot churn per pool. Knowing the normal create/destroy rate makes an anomalous burst of destroys stand out instead of blending in.
5. Know your buffer size. A pool created small and grown large may hold only hours of history under snapshot churn. That tells you how stale a polled scrape is allowed to be, and it is usually “not at all”.

Signals to watch in production

How Netdata helps

• Netdata’s ZFS collectors already surface the operational side of the same story: pool health state, capacity, per-vdev error counters, and scrub status, so a history event like a zpool clear, replace, or offline can be correlated against the state change it produced.
• Capacity and snapshot-space trending gives the “before” picture when history shows a mass snapshot destruction: you can see whether the destroy freed the space someone claimed it would, or whether a freeing backlog is still being reclaimed.
• Because history events arrive through syslog once the ZED exclusion is removed, they land in the same timeline as the rest of the node’s metrics, which is what makes “someone disabled sync at 14:03 and the database corruption report came at 14:40” a five-minute correlation instead of a week-long archaeology project.
• Alerting on pool state transitions (ONLINE to DEGRADED, unexpected exports affecting availability) closes the loop: the metric alert tells you something changed, and the shipped history tells you whether a human changed it.

Related guides

• ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM
• ZFS zfs_arc_max: capping the ARC without starving read performance
• ZFS ARC and the OOM killer: applications killed while the cache will not shrink fast enough
• ZFS ARC shrinking below c_max: reading memory pressure before latency hits
• ZFS ARC using all memory: the Linux default that eats your RAM
• ZFS cannot destroy dataset is busy: clones, holds, and mounted filesystems
• ZFS cannot import pool: missing devices, a damaged label, and root-pool boot failure
• ZFS capacity planning: runway estimation before the pool fills
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS deadman events: hung I/O and a stalled pool sync
• ZFS dedup memory exhaustion: when the DDT outgrows ARC and the pool crawls