$ guides / traefik / traefik-certificate-expired ▌

Operations Guides

Traefik certificate expired: when ACME renewal has been failing silently

Browsers are rejecting your site with a certificate error, and Traefik is the TLS terminator. The process is up, /ping returns 200, traffic flows on port 80, and every HTTPS client reports an expired or expiring certificate. This is the signature of silent ACME renewal failure.

The nasty part is the timeline. Traefik attempts renewal 30 days before expiry. For a 90-day Let’s Encrypt certificate, that means renewal has been attempted since day 60. If a certificate is 7 days from expiry, renewal has already been failing for roughly 53 days. There is no Prometheus metric for ACME failure. The only metric you get is the consequence: traefik_tls_certs_not_after sliding toward the current time while nothing alerts.

This guide covers confirming the failure, finding the cause in the logs, forcing renewal, and closing the monitoring gap so you catch the next failure at day 61 instead of day 89.

What this means

Traefik’s ACME resolver runs as a background process. It renews certificates, writes them to its store (acme.json or a KV backend), and the TLS handshake layer keeps serving whatever certificate it has. Serving and renewing are decoupled: Traefik will serve an expiring, and eventually expired, certificate while the renewal loop errors in the background. If Let’s Encrypt is unreachable, Traefik falls back to previously generated certificates, then expired ones, then any provided certificates. There is no circuit breaker that stops serving a dying cert.

The failure is silent by design: renewal errors go to the log, not to any metric. The system “works” right up until client trust stores reject it.

flowchart LR
  A["Day 0: cert issued, 90-day validity"] --> B["Day 60: renewal window opens"]
  B --> C["Renewal failing, retries log errors only"]
  C --> D["Day 83: less than 7 days left"]
  D --> E["Day 90: expiry, browsers reject"]
  B -.->|healthy path| F["Renewal succeeds, not_after jumps forward"]

The moment you are debugging an expired certificate, you are at the end of a multi-week failure. Your job is twofold: fix the renewal now, and move detection to the start of the window.

Common causes

Cause	What it looks like	First thing to check
HTTP-01 challenge blocked	Firewall, security group, or another service took port 80; challenge never completes	Is port 80 reachable from the internet and answered by Traefik?
DNS-01 challenge failure	Propagation timeouts; errors like `propagation: time limit exceeded` in logs	Is the DNS provider API credential still valid and the env var named correctly?
Rotated DNS API token	Renewal worked for months, broke after a credential rotation, no config change on your side	Traefik uses Lego env var names; verify them against the Lego provider docs
Let’s Encrypt rate limit	`urn:ietf:params:acme:error:rateLimited` in logs; often after frequent restarts or many domains	Count recent issuances; limits include 50 certs per registered domain per week and 5 duplicate certs per week
`acme.json` permissions not 600	Traefik skips the resolver or logs “permissions … are too open”; renewal silently stops	`ls -l` on `acme.json`, must be exactly 600
`acme.json` corrupted	Hard failure at startup; mid-operation corruption leaves in-memory certs serving while renewal silently fails	Parse the file as JSON; check startup logs
Staging CA left configured	`caServer` points at Let’s Encrypt staging; renewals “succeed” but browsers reject the Fake LE chain	Check the `caServer` value and the cert issuer
“No ACME certificate generation required”	DEBUG-level log line; Traefik found a matching (possibly stale or wildcard) cert in the store and skipped	Inspect `acme.json` for stale entries covering the domain
HA ACME lock stuck	Multi-instance with Consul/etcd lock; one instance crashed holding the lock; logs show lock acquisition failure	Inspect and clear the stale lock in the KV store

Quick checks

All read-only and safe to run during the incident.

# What cert is actually being served, and when does it expire?
echo | openssl s_client -servername your.domain.com -connect your.domain.com:443 2>/dev/null \
  | openssl x509 -noout -dates -issuer -serial

# What does Traefik think it has? Expiry timestamps per CN/serial/SANs.
curl -s http://localhost:8080/metrics | grep traefik_tls_certs_not_after

# Seconds remaining per certificate (run in Prometheus):
# traefik_tls_certs_not_after - time()

# acme.json permissions: must be exactly 600
ls -l /path/to/acme.json

# acme.json parses as valid JSON
python3 -c "import json; json.load(open('/path/to/acme.json'))" && echo OK

# Recent ACME errors in the logs (pick your variant)
grep -iE 'acme|renew|challenge' /var/log/traefik/traefik.log | tail -50
# docker logs traefik --since 168h 2>&1 | grep -iE 'acme|renew'
# kubectl logs deploy/traefik -n traefik --since=168h | grep -iE 'acme|renew'

To inspect every certificate stored in acme.json and its real expiry (requires the Python cryptography package):

python3 -c "
import json, base64
from cryptography import x509
with open('/path/to/acme.json') as f:
    data = json.load(f)
for resolver in data:
    for cert in data[resolver].get('Certificates', []):
        c = x509.load_pem_x509_certificate(base64.b64decode(cert['certificate']))
        print(cert['domain']['main'], 'expires', c.not_valid_after_utc)
"

How to diagnose it

Confirm which certificate is being served. The openssl s_client check above shows the live cert’s expiry, issuer, and serial. If the issuer is a staging CA (“Fake LE Intermediate” or similar), the resolver is pointed at the staging server and every “successful” renewal has been producing unusable certs.
Check what the store holds. Compare the served serial against the serials in traefik_tls_certs_not_after and in acme.json. If the store contains a newer, valid cert that is not being served, the problem is cert selection, not renewal. If the store only has the expiring cert, renewal itself is broken.
Read the renewal errors. Search the logs for “Error renewing certificate” and ACME challenge errors. The error string routes you:
- rateLimited means Let’s Encrypt quota; no amount of restarting helps. Wait out the window or deploy a stop-gap cert.
- propagation: time limit exceeded or DNS errors mean DNS-01 is broken: credentials, resolver reachability, or the DNS provider API.
- Connection or timeout errors on the HTTP-01 path mean port 80 is not reaching Traefik.
- “permissions … are too open” means acme.json mode is wrong.
- “unable to acquire lock” in HA setups means a stale distributed lock.
Verify the challenge path manually. For HTTP-01, confirm port 80 is reachable from outside and terminates on this Traefik instance. For DNS-01, confirm the DNS provider credentials work and that Traefik’s environment uses the variable names Lego expects, which are not Traefik-specific names. The Lego provider documentation is the authoritative list.
Rule out the store. If the logs show nothing useful, check acme.json permissions and integrity. Also look for stale entries: Traefik renews certificates that are no longer referenced, which burns rate limit quota and can mask the real problem.
Check the DEBUG-level skip. If “No ACME certificate generation required for domains” appears only at DEBUG level, Traefik found an existing certificate in the store it considers a match (including a wildcard or stale entry covering the hostname) and skipped issuance entirely. Cleaning the stale entry or correcting the router’s certificate resolver is the fix.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`traefik_tls_certs_not_after`	The only metric in the whole failure chain; expiry timestamp per `cn`, `serial`, `sans`	Within 30 days: renewal window open. Within 14 days: renewal has failed. Within 7 days: it has failed for ~53 days
Certificate serial change over time	A successful renewal produces a new serial and a jump forward in `not_after`	Same serial for weeks inside the renewal window means renewal is not happening
ACME error lines in logs	The only place the failure cause exists	Any “Error renewing certificate”, challenge error, or `rateLimited` line
External TLS probes per production hostname	`traefik_tls_certs_not_after` covers all certs in the store, including dormant ones; it cannot tell you what is actually served	Probe shows an actively served cert nearing expiry: page-worthy
`traefik_entrypoint_requests_total` on the HTTPS entrypoint	Collapse after expiry confirms user impact	Sudden drop on 443 while port 80 is normal

One caveat on traefik_tls_certs_not_after: after a successful renewal, the old certificate’s series may not be cleaned up, so Prometheus can show multiple serials for the same CN and produce false-positive expiry alerts. A common workaround is alerting on topk(1, traefik_tls_certs_not_after) - time() so only the newest serial counts. As Traefik v3.7.12, issue #8606 is still open and PR #11659 is not merged.

Fixes

Challenge path broken (HTTP-01 or DNS-01)

Restore the challenge path and let the retry loop recover on its own; Traefik keeps retrying. For HTTP-01, fix the firewall or the process squatting on port 80. For DNS-01, fix the credentials using the exact env var names from the Lego provider docs, then restart Traefik if env vars changed. If propagation checks are flaky against your DNS, look at the DNS challenge propagation options for your Traefik version. In v3.3 the older delayBeforeCheck and disablePropagationCheck keys were deprecated in favor of propagation.delayBeforeChecks and propagation.disableChecks.

Rate limited by Let’s Encrypt

You cannot force your way through a rate limit. If the certificate has days left, wait out the window and stop whatever is burning quota: frequent restarts triggering re-issuance, staging or test domains on the production resolver, or stale certs in acme.json that Traefik keeps renewing. If the cert expires before the window resets, deploy a stop-gap certificate from another CA or a manually obtained cert as a provided certificate, then fix the quota consumption.

`acme.json` permissions or corruption

chmod 600 /path/to/acme.json

If the file is corrupted, back it up, remove it, and restart Traefik to trigger fresh issuance. This is disruptive: Traefik will re-issue every certificate, and fresh issuance for many domains at once can itself hit rate limits, so prune dead domains from your configuration first. Note the behavioral asymmetry: a corrupted acme.json is a hard failure at startup (visible), but corruption mid-operation fails silently while in-memory certs keep serving.

Stale store entries and the silent skip

If Traefik logs “No ACME certificate generation required” while the cert expires, remove the stale or overly broad entry (for example an old wildcard) from acme.json so the resolver no longer considers the domain covered. Back up the file before editing, then restart Traefik: the local ACME store is read once into memory, so in-process resolver passes do not reread manually edited acme.json.

HA lock contention

Traefik Proxy 2/3 has no built-in Consul or etcd ACME lock, so a stale built-in KV lock is not a cause here. Run one ACME-capable instance or use a product designed for ACME HA; if a custom or legacy v1 lock backend is involved, clear the stale lock in that backend.

Emergency stop-gap

If the cert expires within 24 hours and the cause is not yet fixed, deploy a manually obtained certificate as a provided certificate. This restores service immediately and decouples recovery from the ACME investigation.

Prevention

Alert on the renewal window, not the expiry. Page at <7 days only with a synthetic probe confirming the cert is actively served; ticket at <14 days; track at <30 days, because 30 days is when renewal should have succeeded. A serial that has not changed by day 75 of a 90-day cert is a failed renewal.
Log-based alerting on ACME errors. Since no metric exists for renewal failure, ship Traefik logs somewhere you can alert on “Error renewing certificate”, challenge errors, and rateLimited.
Synthetic TLS probes per production hostname. This closes the dormant-cert false-positive gap and catches cert selection problems that store metrics cannot see.
Protect the store. Keep acme.json at 600, back it up, and monitor it for parse failures. Prune certificates for decommissioned domains so Traefik stops renewing them and burning quota.
Track rate limit headroom. In dynamic environments where CI/CD creates subdomains, certificate issuance count is a capacity resource. Stay well under 50 per registered domain per week.
Test renewal after upgrades. The v2 to v3 migration changed rule syntax (for example Host() accepting a single domain), which broke ACME detection for some operators. After any Traefik upgrade, verify renewal still works before the old cert approaches its window.

How Netdata helps

Certificate countdown as a first-class chart. Netdata collects traefik_tls_certs_not_after and lets you alert on time-to-expiry, so the renewal window itself becomes the alerting threshold instead of browser complaints.
Per-second correlation with entrypoint traffic. When a cert expires, correlating the expiry timestamp with the HTTPS entrypoint request rate shows the exact moment client impact started and confirms recovery after the fix.
Log and metric correlation on one host. Netdata’s per-host view puts Traefik metrics next to system state (restarts, disk, FD usage), which matters for causes like a volume remount resetting acme.json permissions or frequent restarts driving rate limits.
Restart visibility. process_start_time_seconds alongside cert metrics reveals the “Traefik restarts, re-requests certs, hits rate limits” loop that quietly produces this failure.
Anomaly detection on the expiry series. A not_after value that keeps approaching now without ever jumping forward is the pattern ML-based anomaly detection flags, without hand-tuning a per-domain threshold.