ZFS encryption key not loaded: cannot mount an encrypted dataset

You rebooted a host, imported a pool, or tried to mount a dataset, and ZFS refused with a variation of cannot mount 'tank/secure': encryption key not loaded. The pool itself is ONLINE, zpool status is clean, and the data is intact. The dataset is simply locked: until the encryption key is loaded into the kernel, ZFS cannot decrypt the dataset’s metadata well enough to mount it.

This is one of the more benign ZFS errors, but it causes real outages because it shows up at the worst times: after an unplanned reboot, during a failover to a standby host, or after a zfs receive on a backup server nobody has logged into since. The fix is one command. Making sure you never have to type it at 3 a.m. takes a bit more work.

Two distinct error strings bring operators here, and they mean different things. encryption key not loaded means no key is in memory at all. Key load error: Incorrect key provided for '<dataset>' means you tried to load one and the passphrase or key material was wrong. The diagnostic path below covers both.

What this means

ZFS native encryption wraps each encrypted dataset’s data with a master key, and that master key is itself wrapped by a user key derived from a passphrase or read from a key file. The wrapped master key lives on disk; the user key does not, unless you stored it in a file yourself. When the system boots, nothing is loaded. The pool imports fine, the dataset shows up in zfs list, but keystatus reports unavailable and any mount attempt fails.

Key loading and mounting are separate operations. zfs load-key puts the key into memory and flips keystatus to available, but it does not mount the dataset. You still need zfs mount (or use zfs mount -l, which loads the key and mounts in one step). The reverse is also true: zfs unload-key locks the dataset, but only succeeds once the dataset is unmounted. Operators routinely run load-key, see keystatus: available, and then wonder why the mountpoint is still empty.

flowchart TD
  A[mount fails: encryption key not loaded] --> B[Check keystatus]
  B -->|unavailable| C[Check keylocation]
  B -->|available| H[Run zfs mount - key is loaded, dataset just not mounted]
  C -->|prompt| D[zfs load-key dataset, enter passphrase]
  C -->|file://path| E[Verify key file exists and is readable]
  E -->|file missing| F[Restore key file or load-key -L with alternate location]
  E -->|file present| D
  D -->|Incorrect key provided| G[Wrong passphrase or wrong key file - verify before retrying]
  D -->|success| H

Common causes

Quick checks

All of these are read-only and safe to run on a production system.

# 1. Key state and where ZFS expects the key to come from
zfs get keystatus,keylocation <dataset>

# 2. Which dataset actually holds the key (the encryption root)
zfs get encryptionroot <dataset>

# 3. Key state for every dataset in the pool at once
zfs get keystatus -r <pool> | grep -v -- "-"

# 4. Is the dataset mounted, and where does ZFS think it should be?
zfs get mounted,mountpoint <dataset>

# 5. Key format (passphrase vs hex/raw key file)
zfs get keyformat <dataset>

# 6. If keylocation is file://, confirm the file is there and readable
ls -l /path/from/keylocation

# 7. Recent key-related administrative actions on this pool
zpool history <pool> | grep -i key

What you will see: keystatus is either available or unavailable; there is no partial state. encryptionroot tells you which dataset in the hierarchy actually owns the key: children of an encryption root share its key, so you load the key on the root, not on each child. If encryptionroot shows the dataset itself, it is its own root and needs its own key material.

How to diagnose it

1. Confirm the dataset is encrypted and locked. Run zfs get encryption,keystatus,keylocation,encryptionroot <dataset>. If encryption is off, you have a different problem (see the related guide on mount failures). If keystatus is unavailable, continue.

2. Identify the encryption root. If encryptionroot points at a parent dataset, all key operations happen against that parent. Loading the key on the root unlocks every child that inherits from it. Loading against the child directly will not work the way you expect.

3. Read keylocation before touching anything. prompt means ZFS expects interactive input. file:///path means ZFS reads raw or hex key material (or a passphrase) from that path. If the value is prompt but you believe a key file exists, someone changed the property or received the dataset in a way that reset it.

4. If keylocation=file://, verify the file. Check that the path exists, is readable by root, and contains what you expect. A file that was on a now-unmounted filesystem, a removed USB device, or a different host is a classic cause. If the file legitimately lives elsewhere now, override the location at load time with zfs load-key -L file:///new/path <dataset> without changing the property.

5. Load the key. For a passphrase: zfs load-key <encryptionroot> and type it. For non-interactive loading from a file: zfs load-key -L file:///path <encryptionroot>. To load keys for everything in the pool at once: zfs load-key -a.

6. Handle “Incorrect key provided” carefully. This error means the key material was read but did not unwrap the master key. Do not retry in a loop. Verify you are using the passphrase for this encryption root (different roots have different passphrases), that a key file has not been truncated or replaced, and that keyformat matches what you are supplying (a passphrase typed where keyformat=hex expects 32 bytes of hex will fail). Passphrases must be 8 to 512 bytes; hex and raw keys are exactly 32 bytes.

7. Mount after the key is available. Check zfs get keystatus again, then zfs mount <dataset> or zfs mount -a. Alternatively, zfs mount -l <dataset> prompts for the key and mounts in one step. If the key loaded but the mount still fails, the problem has moved on from encryption to a normal mount problem: check the mountpoint path, conflicting mounts, and canmount.

8. If this was unexpected, audit. A dataset that was unlocked and is now locked without a reboot means someone ran zfs unload-key, exported the pool, or the dataset was never re-keyed after an event. zpool history <pool> | grep -i key shows load, unload, and change-key operations with timestamps. Treat unexplained key events as a security signal, not just an operational nuisance.

Metrics and signals to monitor

Fixes

One-time unlock right now

Load the key on the encryption root and mount:

# Load the key (prompts for passphrase), then mount
zfs load-key tank/secure
zfs mount tank/secure

# Or in one step
zfs mount -l tank/secure

For a key file stored somewhere other than the configured keylocation:

# Override key location for this load only
zfs load-key -L file:///root/keys/tank-secure.key tank/secure
zfs mount tank/secure

This is the whole fix for the immediate incident. The tradeoff is that it is manual, and it will be needed again after every reboot.

Survive reboots: key file with automatic loading

If the dataset must come up unattended, set keylocation to a file that exists at boot time:

# Point the dataset at a key file, then verify a load works
zfs set keylocation=file:///root/keys/tank-secure.key tank/secure
zfs unload-key tank/secure    # requires the dataset to be unmounted
zfs load-key tank/secure      # should succeed with no prompt
zfs mount tank/secure

The tradeoff is real: a key file on the same host as the pool protects against very little. Anyone who takes the disks and the host has both halves. Common mitigations are keeping the key file on the root filesystem of a separate boot device (so the data disks alone are useless), restricting permissions to root-only, and treating the key file as a secret in your backup and configuration management systems. Never store the key file on the encrypted pool itself.

On systemd-based Linux distributions, OpenZFS 2.0 and later ship a mount generator that creates key-loading service units for encryption roots so datasets with keylocation=file:// are unlocked during boot. If automatic loading is not happening, check whether the generator is enabled and whether its cache file is populated; behavior has varied across packaging and releases.

Survive reboots: interactive prompt

If policy forbids key files, keylocation=prompt is correct, but you must accept that boot stalls until someone enters the passphrase, or that an operator runs zfs load-key -a && zfs mount -a after every boot. Document that in the boot runbook. A host that “came back from reboot but the application is down because /data is empty” is this failure mode.

Fixing a wrong keylocation after replication

Datasets received via zfs send can arrive with a keylocation that does not match the receiving host’s layout, and there are known issues where raw sends of individual encrypted sub-filesystems (not the encryption root) can create a new encryption root and set keylocation to prompt on the receive side; OpenZFS issue #15989 remains open. If the receiving side has the right key on the parent, re-inherit it:

# Re-inherit the parent's key (supported since OpenZFS 0.8.0)
zfs change-key -i tank/secure/child

Otherwise set keylocation explicitly on the receiver to a path that exists there.

Rotating or repairing key material

zfs change-key replaces the user key (the passphrase or key file), not the master key that encrypts the data. That distinction matters for two reasons. First, rotating the passphrase is cheap and does not re-encrypt the dataset. Second, the old wrapped master key material may still be recoverable from disk by forensic analysis, so passphrase rotation is not a remediation for fully compromised key material. If you genuinely need new master encryption, the only clean path is zfs send into a freshly encrypted dataset.

Prevention

• Decide the unlock model per encryption root and write it down. Prompt for human-attended systems, key file for unattended ones, and document which is which. Most incidents are a mismatch between what the property says and what the operator assumes.
• Test the boot path, not just the happy path. After setting up encryption, actually reboot the host (or at minimum export and re-import the pool) and confirm the dataset mounts unattended. This is the check nobody runs and everybody regrets.
• Back up key material separately from the pool. A passphrase in your password manager or a key file in your secrets store, referenced by encryption root name. Losing the user key means the data is cryptographically gone; ZFS cannot help you.
• Monitor keystatus for every encrypted dataset. Alert on unavailable outside reboot windows. This catches both failed boot-time loading and unexpected unloads.
• Audit key events. Periodically review zpool history for load-key, unload-key, and change-key operations. An unexpected unload on a production dataset is an availability incident and a security event at the same time.
• Keep clones and children in mind. Clones always use the origin’s key, and encryptionroot, keyformat, keylocation, and pbkdf2iters do not inherit like ordinary properties. Verify key state on clones before assuming they unlock with the parent.

How Netdata helps

• ZFS pool and dataset visibility: Netdata’s ZFS collector tracks pool health, capacity, and dataset state continuously, so a locked dataset shows up as a gap in expected filesystem metrics right after boot rather than when the application team notices.
• Correlation with reboot and import events: Because Netdata keeps per-second history, you can line up the exact moment of the reboot or pool import with when dataset metrics stopped, confirming the key-loading failure window.
• Application-side confirmation: When a locked dataset takes an application down, Netdata’s application and filesystem metrics show the downstream impact (empty mountpoint, failing reads, process errors) in the same dashboard as the pool state, shortening the “is it the app or the storage” loop.
• Alerting on state changes: Netdata alarms on ZFS health and capacity signals can be extended with an external check on zfs get keystatus so an unavailable key pages before users do.

Related guides

• ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM
• ZFS zfs_arc_max: capping the ARC without starving read performance
• ZFS ARC and the OOM killer: applications killed while the cache will not shrink fast enough
• ZFS ARC shrinking below c_max: reading memory pressure before latency hits
• ZFS ARC using all memory: the Linux default that eats your RAM
• ZFS cannot destroy dataset is busy: clones, holds, and mounted filesystems
• ZFS capacity planning: runway estimation before the pool fills
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS deadman events: hung I/O and a stalled pool sync
• ZFS deleted files but no space freed: snapshots holding the blocks
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus