BIND journal (.jnl) corruption: dynamic-update and IXFR failures that block zone load

A zone fails to load after a restart or rndc reload. The named process is running, other zones respond normally, and named-checkzone reports the zone file is syntactically valid. The problem is the journal file (.jnl) alongside the zone file, which records pending dynamic updates and drives IXFR between primary and secondary servers.

Journal corruption has two operational faces. The acute case: named starts but refuses to serve the affected zone, logging “journal rollforward failed: journal out of sync with zone.” The subtle case: IXFR transfers keep failing and falling back to full AXFR. The zone stays current, but each refresh pulls the entire zone over TCP instead of just the diff, wasting bandwidth and CPU.

Both stem from the same root cause: the .jnl file is either internally inconsistent with the zone file, corrupted by an unclean write, or in a format the running named version cannot parse.

What this means

When a zone accepts dynamic updates (via allow-update, update-policy, or RFC 2136 clients) or receives IXFR as a secondary, BIND maintains a journal file in the zone’s working directory. This file is an append-only log of changes applied since the last full write of the zone file. On startup, named loads the zone file, then replays pending journal entries to bring the in-memory zone up to date. If the journal cannot be read or does not align with the zone file, named refuses to load the zone rather than serve inconsistent data.

The critical detail: named starts successfully even when zones fail to load. The process is alive, rndc status reports “running,” and queries to other zones return normally. Only queries to the affected zone fail with REFUSED or SERVFAIL. Process-level health checks will not catch this.

The .jnl format changed in BIND 9.16.12 to support the max-ixfr-ratio option. If named was stopped ungracefully before an upgrade, pending journal entries written in the old format may not be readable by the new version. BIND 9.16.13 auto-upgrades old-format journals on load, but operators running 9.16.12 or 9.17.0 specifically can still hit this.

flowchart TD
    A["Zone fails to load
after restart or reload"] --> B{"Log message?"}
    B -->|"journal rollforward failed"| C["Journal corruption"]
    B -->|"IXFR failed, attempting AXFR"| D["IXFR fallback
(early symptom)"]
    B -->|"No journal mention"| E["Zone file syntax
or permissions"]
    C --> F{"Upgraded to
9.16.12 or 9.17.0?"}
    F -->|"Yes"| G["Format mismatch"]
    F -->|"No"| H["Unclean shutdown
or manual edit"]
    G --> I["named-journalprint -u,
or delete .jnl and restart"]
    H --> J["rndc sync -clean
then rndc reload"]
    D --> J

Common causes

Quick checks

# Check for journal-related errors after startup or reload
journalctl -u named --since "30 min ago" | grep -i "journal\|rollforward\|jnl"

# Check if the affected zone is loaded
rndc zonestatus example.com

# Validate the zone file itself (rules out syntax errors)
named-checkzone example.com /var/named/example.com.zone

# Check for IXFR-to-AXFR fallback (early symptom)
journalctl -u named --since "24 hours ago" | grep -i "IXFR failed.*AXFR"

# Look for the .jnl file alongside the zone file
ls -la /var/named/example.com.zone.jnl 2>/dev/null || echo "NO JNL FILE"

# Check BIND version (relevant for format compatibility)
named -V | head -1

# Check dynamic update activity
# <!-- TODO: verify port. 8653 is non-standard; adjust to match your statistics-channel config -->
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{k}: {v}') for k,v in d.get('nsstats',{}).items() if k.startswith('Update')]"

# Check managed-keys status (if DNSSEC validation is in use)
rndc managed-keys status

# Compare SOA serials between primary and secondary
dig @primary-ip example.com SOA +short
dig @secondary-ip example.com SOA +short

How to diagnose it

1. Confirm the zone is not loaded. Run rndc zonestatus example.com. If the zone failed to load, the command returns an error or shows no serial. Other zones on the same server should respond normally, confirming the problem is zone-specific.

2. Check the logs. Look for one of these patterns:

   • journal rollforward failed: journal out of sync with zone confirms journal corruption.
   • IXFR failed, attempting AXFR confirms IXFR fallback on a secondary. This is the early warning before an acute failure.
   • zone example.com/IN: loading from master file ... failed without a journal mention points to zone file syntax or permission issues instead.

3. Validate the zone file. Run named-checkzone example.com /path/to/zone/file. If it passes, the zone data is valid and the journal is the suspect. If it fails, fix the syntax error first.

4. Check for version-related format issues. Run named -V. If you recently upgraded to 9.16.12 or 9.17.0, old-format .jnl files may be unreadable. BIND 9.16.13 and later auto-upgrade old journals on load.

5. Check for unclean shutdown. Look at dmesg | grep -i oom, systemctl show named --property=NRestarts, and the systemd journal for kill events. rndc stop always flushes pending journal entries to disk. rndc halt does not. SIGTERM (the default signal from systemctl stop) respects the flush-zones-on-shutdown option, which defaults to no; set it to yes in named.conf to make systemd-initiated stops safe for dynamic zones. SIGKILL and OOM kills leave journals in a partial state regardless.

6. Check update activity. Elevated UpdateFail counters indicate broken dynamic update clients or journal write problems. Compare UpdateDone vs UpdateFail ratios.

7. Check IXFR fallback frequency. Search logs for repeated IXFR failed, attempting AXFR patterns. Occasional fallback can happen under normal conditions. Repeated fallback on every transfer cycle indicates journal corruption on the source.

Metrics and signals to monitor

Fixes

Online recovery: rndc sync -clean then rndc reload

The fastest recovery when named is running and the zone is loaded but unhealthy:

# Flush pending journal entries to the zone file and remove the .jnl
rndc sync -clean example.com

# Reload the zone from the now-clean zone file
rndc reload example.com

This flushes the journal into the zone file, removes the .jnl file, and reloads the zone from disk. The zone serial is preserved.

Tradeoff: Any dynamic updates received between the journal corruption and the sync may be lost. If the journal was already unreadable, those updates were already effectively lost.

Offline recovery: delete the .jnl file manually

If rndc sync -clean fails or named cannot process the command (for example, the zone never loaded):

# Stop named cleanly
rndc stop

# Remove the corrupted journal file
rm /var/named/example.com.zone.jnl

# Start named
systemctl start named

After restart, named loads the zone from the zone file without replaying any journal entries. The zone serial reflects the last successful write to disk.

Warning: This loses any dynamic updates that were in the journal but not yet written to the zone file. Coordinate with dynamic update sources (DHCP servers, Kubernetes external-dns, infrastructure automation) to ensure they re-send recent updates.

Manual editing of dynamic zones

If you need to edit a zone file that accepts dynamic updates, you must freeze the zone first:

# Freeze the zone (stops dynamic updates, flushes journal to zone file)
rndc freeze example.com

# Edit the zone file
vi /var/named/example.com.zone

# Increment the serial if your tooling does not do it automatically

# Thaw the zone (resumes dynamic updates, creates a new .jnl)
rndc thaw example.com

Editing a dynamic zone file without rndc freeze is one of the most common causes of journal corruption. The journal and zone file diverge silently, and the next restart fails.

Format conversion with named-journalprint

If the corruption is due to a version format mismatch (upgrading to or from 9.16.12):

# Stop named first (this tool must not run while named is active)
rndc stop

# Upgrade the journal format
named-journalprint -u /var/named/example.com.zone.jnl

<!-- TODO: verify -d flag exists for downgrade. named-journalprint -d may mean debug, not downgrade. -->
# Or downgrade if rolling back
named-journalprint -d /var/named/example.com.zone.jnl

# Start named
systemctl start named

This converts the .jnl file between pre-9.16.12 and post-9.16.12 formats. If conversion fails, delete the .jnl file and restart. The zone file on disk is still valid, and named will create a fresh journal on the next dynamic update.

managed-keys.bind.jnl corruption

If the rollforward error appears for the managed-keys zone rather than a regular zone, DNSSEC validation is at risk. The recovery is the same: stop named, remove the .jnl file, and restart. named will re-establish trust anchors via RFC 5011 on the next startup. This may cause a brief window of elevated ValFail while trust anchors re-converge.

Prevention

• Use rndc stop, not rndc halt. rndc stop always flushes pending journal entries to disk. rndc halt does not. SIGTERM from systemctl stop respects flush-zones-on-shutdown, which defaults to no. Set flush-zones-on-shutdown yes; in named.conf to make systemd-initiated stops safe for dynamic zones.

• Always freeze before editing. Any manual edit to a dynamic zone file must be preceded by rndc freeze and followed by rndc thaw. This is the single most common preventable cause of journal corruption.

• Verify zone loads after every reload. After rndc reload, check rndc zonestatus for each affected zone. A failed zone load is logged but easy to miss if you reload and walk away. See the BIND monitoring checklist for a post-reload verification procedure.

• Be cautious with ixfr-from-differences. This option generates IXFR diffs from zone file changes, but it is resource-intensive and has a known race condition with rndc stop. ISC recommends not enabling it on servers that also handle client queries.

• Plan upgrades around journal state. Before upgrading BIND, especially to 9.16.12 or 9.17.0, run rndc sync -clean on all dynamic zones to flush journals and eliminate old-format files. Then stop with rndc stop before the upgrade.

• Monitor disk space. A full partition prevents journal writes, which can leave .jnl files in a partial state. Monitor the partition holding zone and journal files.

How Netdata helps

• SERVFAIL correlation with restarts. Per-second resolution makes it possible to correlate a QrySERVFAIL spike with a specific restart or reload event, pointing at a zone load failure rather than a network or upstream issue.

• UpdateFail rate tracking. Continuous collection of BIND’s UpdateFail counter surfaces journal write problems before they escalate to a full zone load failure. A rising trend from known update clients is an early warning.

• IXFR-to-AXFR fallback detection. Correlating transfer mode changes with restart or upgrade events helps distinguish journal corruption from TSIG mismatches or network partitions.

• Zone serial divergence. Comparing SOA serials across primary and secondaries over time catches transfer failures that may originate from journal corruption on the source.

Related guides

• How BIND actually works in production: a mental model for operators
• BIND monitoring checklist: the signals every production resolver and authoritative server needs
• named not responding on port 53: total outage versus UDP-works-TCP-fails
• BIND monitoring maturity model: from survival to expert
• BIND cold cache after restart: the warming storm and elevated upstream load
• BIND ’no more recursive clients: quota reached’: the recursive-clients circuit breaker