Traefik acme.json permissions and corruption: renewal silently blocked

Your certificates are approaching expiry and Traefik has not renewed them. There is no alert, no error metric, and depending on how the failure happened, possibly nothing useful in the logs either. The root cause in a large share of these incidents is the ACME storage file itself: acme.json has the wrong permissions, or it has been corrupted by a partial write, a restore, or an out-of-memory event during a write.

This failure mode is dangerous because it is quiet. Traefik keeps serving the certificates it already holds in memory, /ping returns 200, and traffic flows normally right up until a certificate expires. If you are reading this because renewal has stopped working, start with the file, not the ACME account or the challenge configuration.

What this means

Traefik stores all ACME account data and issued certificates in a single JSON file (conventionally acme.json, at whatever path your certificate resolver’s storage field points to). Two distinct failure modes attach to this file:

Permissions. Traefik requires acme.json to have mode 0600. If the file is more permissive, Traefik refuses to read or write it. The common trigger is not an operator running chmod carelessly. It is infrastructure doing it for you: a volume mount that applies its own permissions, a backup restore that recreates the file with default modes, or a filesystem that does not preserve Unix modes on bind mounts. Because Traefik may refuse the file on write rather than at startup, the resolver can stop persisting new certificates while the running process keeps serving the ones it already has. Renewal is silently blocked.

Corruption. A partial write, a filesystem error, or an OOM kill during a write can leave acme.json truncated or invalid. Here the behavior splits by timing:

• If the file is corrupt at startup, Traefik hard-fails. You see this one immediately because the process will not come up cleanly.
• If the file is corrupted mid-operation, Traefik does not crash. In-memory certificates keep serving, and the failure only surfaces on the next restart or the next renewal attempt that needs to write.

flowchart TD
  A["Event: mount, restore, chmod, partial write, OOM during write"] --> B{"acme.json state"}
  B -->|"permissions wider than 600"| C["Traefik refuses to read/write storage"]
  B -->|"corrupt at startup"| D["Hard fail at boot"]
  B -->|"corrupt mid-operation"| E["Silent failure, in-memory certs keep serving"]
  C --> F["Renewal never persists"]
  E --> F
  F --> G["traefik_tls_certs_not_after approaches now"]
  G --> H["Certs expire, clients see TLS errors"]
  D --> I["Visible immediately on restart"]

The operational consequence: the only built-in signal you get is traefik_tls_certs_not_after drifting toward the present. There are no Prometheus metrics for ACME renewal failures or rate limit hits. Everything else lives in logs, or nowhere.

Common causes

One version note: Traefik’s issue tracker documents a shutdown path where the process could truncate acme.json while writing it, producing invalid JSON on the next start. A graceful-shutdown fix for the write path was merged for the v2.11 line. If you are on an older v2 release and manage many certificates, treat unexpected truncation during restarts as a known bug class and plan an upgrade.

Quick checks

All of these are read-only and safe to run during an incident.

# 1. File mode and ownership. Must be 600, owned by the Traefik process user.
stat -c '%a %U:%G %n' /path/to/acme.json

# 2. JSON validity. A truncated file fails here immediately.
jq empty /path/to/acme.json && echo "valid JSON" || echo "CORRUPT"

# 3. File size sanity. A file that suddenly shrank is suspect.
ls -l /path/to/acme.json

# 4. Permission errors in Traefik logs.
# Look for the "too open" refusal and ACME read/write errors.
grep -iE "permissions.*too open|can't read acme|error renewing certificate" /var/log/traefik/traefik.log

# 5. Certificate expiry from metrics. Renewal failing means this drifts toward now.
curl -s http://localhost:8080/metrics | grep traefik_tls_certs_not_after

# 6. Restart history. Mid-operation corruption plus restarts is the dangerous combination.
# Kubernetes:
kubectl get pod -n <ns> <traefik-pod> -o jsonpath='{.status.containerStatuses[*].restartCount}'
# Bare metal / systemd:
systemctl status traefik | grep -i active

Two notes on check 4. The permission refusal log line names the offending mode explicitly (“permissions … are too open, please use 600”), which makes it the fastest confirmation path. And if the file lives inside a container, run stat from inside the container too: what matters is the mode Traefik sees, not the mode on the host.

How to diagnose it

1. Confirm renewal is actually broken. Pull traefik_tls_certs_not_after for the affected CNs. Let’s Encrypt certificates are valid for 90 days and Traefik attempts renewal 30 days before expiry. A certificate inside 30 days with an unchanged serial means renewal has already failed at least once. A certificate inside 7 days means it has been failing for weeks.

2. Check the file mode. If stat shows anything other than 600, you have a strong candidate. Work out what touched the file: a restore job, a volume driver, an init container, a host-side backup tool. The mode rarely changes by itself.

3. Check ownership. The file must be writable by the user the Traefik process runs as. In containerized deployments the process often runs as a non-root UID, and a file created on the host as root can be unwritable to it. Compare the file owner against the process UID (ps -o user= -p $(pgrep traefik) on bare metal, or inspect the container’s security context).

4. Validate the JSON. If the mode is fine, parse the file. A corrupt file at startup produces an obvious boot failure; a corrupt file discovered while Traefik is running means you are in the silent-failure window and the in-memory certificates are the only thing keeping TLS up.

5. Correlate with restarts. If the file is corrupt and the pod or process has restarted recently, check whether the restart was an OOM kill. Memory exhaustion during the storage write is a known corruption source. go_memstats_heap_inuse_bytes trending toward the container limit before the restart would corroborate it.

6. Check for rate limit damage. If Traefik has been restarting with unwritable or missing storage, it may have been requesting fresh certificates on every boot. A crash-looping instance can burn through Let’s Encrypt’s per-domain weekly issuance limits quickly. If logs show repeated successful issuances for the same domains, assume you are now rate-limited and factor that into the fix: repairing the file will not immediately restore issuance if the weekly window is exhausted.

Metrics and signals to monitor

The last row is the uncomfortable one. There is no built-in integrity monitoring for acme.json. If you want early warning, you have to build it: a file-mode check and a JSON-validity check in your node-level monitoring, and a checksum alert that distinguishes “changed because a cert renewed” from “changed unexpectedly.”

Fixes

Wrong permissions or ownership

# Restore the required mode. Safe, immediate, no restart needed in most cases.
chmod 600 /path/to/acme.json

# Fix ownership if the process user differs. Substitute your Traefik UID.
chown <traefik-uid>:<traefik-gid> /path/to/acme.json

Then watch the logs for the next renewal attempt. If Traefik already refused the file at startup, a restart of the Traefik process is required for the resolver to re-initialize against the corrected file. Schedule it; do not restart reflexively, since a restart with any other unresolved storage issue triggers fresh issuance attempts.

If a volume driver or restore pipeline keeps resetting the mode, fix it there. In Kubernetes, an init container that chmods and chowns the file before the main container starts is the standard workaround for storage drivers that force wider modes. On Docker Desktop with host bind mounts that cannot express 0600, move acme.json to a named volume instead of a bind mount.

Corrupt acme.json

Back up before you touch anything:

# Preserve evidence and any salvageable account data before recreating.
cp /path/to/acme.json /path/to/acme.json.corrupt.$(date +%Y%m%d%H%M%S)

Then remove or replace the file with an empty structure (or let Traefik create it), set mode 600, and restart. Understand the cost first: recreating the storage triggers fresh issuance for every certificate the resolver manages. That means real ACME orders against your rate limits. If you manage many domains on one file, or you have already burned issuance budget through restart loops, fresh orders may be rejected for days. Where possible, extract still-valid certificates from the corrupt file first and confirm your challenge path works before forcing mass re-issuance.

If Traefik is currently serving from memory with a corrupt file on disk, you have a window: traffic is fine until the next restart. Use it to back up, verify challenge reachability (port 80 for HTTP-01, DNS provider credentials for DNS-01), and plan the restart deliberately.

Rate limit exhaustion

If repeated re-issuance has hit Let’s Encrypt limits, there is no fix inside Traefik. You wait out the window. Prevent recurrence by making the storage durable (below) so re-issuance is never triggered spuriously, and keep normal consumption well under the limits so an emergency re-issuance has headroom.

Prevention

• Persist acme.json on storage that preserves Unix modes. Named volumes or PVs with a driver that honors 0600. Never a bind mount that flattens permissions.
• Pin the mode at deploy time. An init container or provisioning step that enforces chmod 600 and correct ownership on every deploy absorbs restore and mount drift.
• Back up acme.json with the file mode intact. A restore that recreates the file with default permissions recreates this incident. Verify mode as part of the restore runbook.
• Use atomic writes anywhere you write config or state files Traefik watches. Write to a temp file, then rename. The same partial-write risk applies to the file provider: a half-written dynamic config picked up by watch: true causes reload failures.
• Alert on the countdown, not the event. Alert when traefik_tls_certs_not_after for any actively-served cert drops under 14 days with no serial change. By 7 days you are doing emergency work.
• Add an external integrity check. Mode, JSON validity, and checksum of acme.json on a schedule. This closes the gap Traefik leaves open.
• Bound restarts. A crash-looping Traefik with broken storage is a rate-limit cannon. Fix crash loops before they become an issuance storm.

How Netdata helps

• Netdata’s Traefik collector exposes traefik_tls_certs_not_after per certificate, so renewal failure shows up as a visible countdown long before expiry, which is the earliest built-in signal this failure mode produces.
• Process metrics (start time, restart detection) correlate a corrupted-on-disk state with the restart that will hard-fail on it, so you can sequence the fix before the process bounces.
• Go runtime metrics (go_memstats_heap_inuse_bytes, RSS against container limits) surface the memory pressure that causes OOM-during-write corruption in the first place.
• Per-second metric correlation helps you line up a permission-refusal log line with the deploy, restore, or mount event that changed the file mode.
• Because Traefik exports nothing about acme.json itself, pairing node-level file and systemd monitoring with the Traefik collector closes the integrity gap the application leaves open.

Related guides

• Traefik certificate expired: when ACME renewal has been failing silently
• Traefik config last reload success: monitoring configuration freshness
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors