Browsers are rejecting your site with a certificate error, and Traefik is the TLS terminator. The process is up, /ping returns 200, traffic flows on port 80, and every HTTPS client reports an expired or expiring certificate. This is the signature of silent ACME renewal failure.

The nasty part is the timeline. Traefik attempts renewal 30 days before expiry. For a 90-day Let’s Encrypt certificate, that means renewal has been attempted since day 60. If a certificate is 7 days from expiry, renewal has already been failing for roughly 53 days. There is no Prometheus metric for ACME failure. The only metric you get is the consequence: traefik_tls_certs_not_after sliding toward the current time while nothing alerts.

This guide covers confirming the failure, finding the cause in the logs, forcing renewal, and closing the monitoring gap so you catch the next failure at day 61 instead of day 89.

What this means

Traefik’s ACME resolver runs as a background process. It renews certificates, writes them to its store (acme.json or a KV backend), and the TLS handshake layer keeps serving whatever certificate it has. Serving and renewing are decoupled: Traefik will serve an expiring, and eventually expired, certificate while the renewal loop errors in the background. If Let’s Encrypt is unreachable, Traefik falls back to previously generated certificates, then expired ones, then any provided certificates. There is no circuit breaker that stops serving a dying cert.

The failure is silent by design: renewal errors go to the log, not to any metric. The system “works” right up until client trust stores reject it.

flowchart LR
  A["Day 0: cert issued, 90-day validity"] --> B["Day 60: renewal window opens"]
  B --> C["Renewal failing, retries log errors only"]
  C --> D["Day 83: less than 7 days left"]
  D --> E["Day 90: expiry, browsers reject"]
  B -.->|healthy path| F["Renewal succeeds, not_after jumps forward"]

The moment you are debugging an expired certificate, you are at the end of a multi-week failure. Your job is twofold: fix the renewal now, and move detection to the start of the window.

Common causes

CauseWhat it looks likeFirst thing to check
HTTP-01 challenge blockedFirewall, security group, or another service took port 80; challenge never completesIs port 80 reachable from the internet and answered by Traefik?
DNS-01 challenge failurePropagation timeouts; errors like propagation: time limit exceeded in logsIs the DNS provider API credential still valid and the env var named correctly?
Rotated DNS API tokenRenewal worked for months, broke after a credential rotation, no config change on your sideTraefik uses Lego env var names; verify them against the Lego provider docs
Let’s Encrypt rate limiturn:ietf:params:acme:error:rateLimited in logs; often after frequent restarts or many domainsCount recent issuances; limits include 50 certs per registered domain per week and 5 duplicate certs per week
acme.json permissions not 600Traefik skips the resolver or logs “permissions … are too open”; renewal silently stopsls -l on acme.json, must be exactly 600
acme.json corruptedHard failure at startup; mid-operation corruption leaves in-memory certs serving while renewal silently failsParse the file as JSON; check startup logs
Staging CA left configuredcaServer points at Let’s Encrypt staging; renewals “succeed” but browsers reject the Fake LE chainCheck the caServer value and the cert issuer
“No ACME certificate generation required”DEBUG-level log line; Traefik found a matching (possibly stale or wildcard) cert in the store and skippedInspect acme.json for stale entries covering the domain
HA ACME lock stuckMulti-instance with Consul/etcd lock; one instance crashed holding the lock; logs show lock acquisition failureInspect and clear the stale lock in the KV store

Quick checks

All read-only and safe to run during the incident.

# What cert is actually being served, and when does it expire?
echo | openssl s_client -servername your.domain.com -connect your.domain.com:443 2>/dev/null \
  | openssl x509 -noout -dates -issuer -serial

# What does Traefik think it has? Expiry timestamps per CN/serial/SANs.
curl -s http://localhost:8080/metrics | grep traefik_tls_certs_not_after

# Seconds remaining per certificate (run in Prometheus):
# traefik_tls_certs_not_after - time()

# acme.json permissions: must be exactly 600
ls -l /path/to/acme.json

# acme.json parses as valid JSON
python3 -c "import json; json.load(open('/path/to/acme.json'))" && echo OK

# Recent ACME errors in the logs (pick your variant)
grep -iE 'acme|renew|challenge' /var/log/traefik/traefik.log | tail -50
# docker logs traefik --since 168h 2>&1 | grep -iE 'acme|renew'
# kubectl logs deploy/traefik -n traefik --since=168h | grep -iE 'acme|renew'

To inspect every certificate stored in acme.json and its real expiry (requires the Python cryptography package):

python3 -c "
import json, base64
from cryptography import x509
with open('/path/to/acme.json') as f:
    data = json.load(f)
for resolver in data:
    for cert in data[resolver].get('Certificates', []):
        c = x509.load_pem_x509_certificate(base64.b64decode(cert['certificate']))
        print(cert['domain']['main'], 'expires', c.not_valid_after_utc)
"

How to diagnose it

  1. Confirm which certificate is being served. The openssl s_client check above shows the live cert’s expiry, issuer, and serial. If the issuer is a staging CA (“Fake LE Intermediate” or similar), the resolver is pointed at the staging server and every “successful” renewal has been producing unusable certs.

  2. Check what the store holds. Compare the served serial against the serials in traefik_tls_certs_not_after and in acme.json. If the store contains a newer, valid cert that is not being served, the problem is cert selection, not renewal. If the store only has the expiring cert, renewal itself is broken.

  3. Read the renewal errors. Search the logs for “Error renewing certificate” and ACME challenge errors. The error string routes you:

    • rateLimited means Let’s Encrypt quota; no amount of restarting helps. Wait out the window or deploy a stop-gap cert.
    • propagation: time limit exceeded or DNS errors mean DNS-01 is broken: credentials, resolver reachability, or the DNS provider API.
    • Connection or timeout errors on the HTTP-01 path mean port 80 is not reaching Traefik.
    • “permissions … are too open” means acme.json mode is wrong.
    • “unable to acquire lock” in HA setups means a stale distributed lock.

  4. Verify the challenge path manually. For HTTP-01, confirm port 80 is reachable from outside and terminates on this Traefik instance. For DNS-01, confirm the DNS provider credentials work and that Traefik’s environment uses the variable names Lego expects, which are not Traefik-specific names. The Lego provider documentation is the authoritative list.

  5. Rule out the store. If the logs show nothing useful, check acme.json permissions and integrity. Also look for stale entries: Traefik renews certificates that are no longer referenced, which burns rate limit quota and can mask the real problem.

  6. Check the DEBUG-level skip. If “No ACME certificate generation required for domains” appears only at DEBUG level, Traefik found an existing certificate in the store it considers a match (including a wildcard or stale entry covering the hostname) and skipped issuance entirely. Cleaning the stale entry or correcting the router’s certificate resolver is the fix.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
traefik_tls_certs_not_afterThe only metric in the whole failure chain; expiry timestamp per cn, serial, sansWithin 30 days: renewal window open. Within 14 days: renewal has failed. Within 7 days: it has failed for ~53 days
Certificate serial change over timeA successful renewal produces a new serial and a jump forward in not_afterSame serial for weeks inside the renewal window means renewal is not happening
ACME error lines in logsThe only place the failure cause existsAny “Error renewing certificate”, challenge error, or rateLimited line
External TLS probes per production hostnametraefik_tls_certs_not_after covers all certs in the store, including dormant ones; it cannot tell you what is actually servedProbe shows an actively served cert nearing expiry: page-worthy
traefik_entrypoint_requests_total on the HTTPS entrypointCollapse after expiry confirms user impactSudden drop on 443 while port 80 is normal

One caveat on traefik_tls_certs_not_after: after a successful renewal, the old certificate’s series may not be cleaned up, so Prometheus can show multiple serials for the same CN and produce false-positive expiry alerts. A common workaround is alerting on topk(1, traefik_tls_certs_not_after) - time() so only the newest serial counts.

Fixes

Challenge path broken (HTTP-01 or DNS-01)

Restore the challenge path and let the retry loop recover on its own; Traefik keeps retrying. For HTTP-01, fix the firewall or the process squatting on port 80. For DNS-01, fix the credentials using the exact env var names from the Lego provider docs, then restart Traefik if env vars changed. If propagation checks are flaky against your DNS, look at the DNS challenge propagation options for your Traefik version. In v3.3 the older delaybeforecheck and disablepropagationcheck keys were deprecated in favor of propagation.delayBeforeChecks and propagation.disableChecks.

Rate limited by Let’s Encrypt

You cannot force your way through a rate limit. If the certificate has days left, wait out the window and stop whatever is burning quota: frequent restarts triggering re-issuance, staging or test domains on the production resolver, or stale certs in acme.json that Traefik keeps renewing. If the cert expires before the window resets, deploy a stop-gap certificate from another CA or a manually obtained cert as a provided certificate, then fix the quota consumption.

acme.json permissions or corruption

chmod 600 /path/to/acme.json

If the file is corrupted, back it up, remove it, and restart Traefik to trigger fresh issuance. This is disruptive: Traefik will re-issue every certificate, and fresh issuance for many domains at once can itself hit rate limits, so prune dead domains from your configuration first. Note the behavioral asymmetry: a corrupted acme.json is a hard failure at startup (visible), but corruption mid-operation fails silently while in-memory certs keep serving.

Stale store entries and the silent skip

If Traefik logs “No ACME certificate generation required” while the cert expires, remove the stale or overly broad entry (for example an old wildcard) from acme.json so the resolver no longer considers the domain covered. Back up the file before editing, and restart or wait for the next resolver pass after changing it.

HA lock contention

In multi-instance setups with Consul or etcd distributed locking, a crashed instance can leave the ACME lock held. Clear the stale lock in the KV store and consider shorter lock TTLs.

Emergency stop-gap

If the cert expires within 24 hours and the cause is not yet fixed, deploy a manually obtained certificate as a provided certificate. This restores service immediately and decouples recovery from the ACME investigation.

Prevention

  • Alert on the renewal window, not the expiry. Page at <7 days only with a synthetic probe confirming the cert is actively served; ticket at <14 days; track at <30 days, because 30 days is when renewal should have succeeded. A serial that has not changed by day 75 of a 90-day cert is a failed renewal.
  • Log-based alerting on ACME errors. Since no metric exists for renewal failure, ship Traefik logs somewhere you can alert on “Error renewing certificate”, challenge errors, and rateLimited.
  • Synthetic TLS probes per production hostname. This closes the dormant-cert false-positive gap and catches cert selection problems that store metrics cannot see.
  • Protect the store. Keep acme.json at 600, back it up, and monitor it for parse failures. Prune certificates for decommissioned domains so Traefik stops renewing them and burning quota.
  • Track rate limit headroom. In dynamic environments where CI/CD creates subdomains, certificate issuance count is a capacity resource. Stay well under 50 per registered domain per week.
  • Test renewal after upgrades. The v2 to v3 migration changed rule syntax (for example Host() accepting a single domain), which broke ACME detection for some operators. After any Traefik upgrade, verify renewal still works before the old cert approaches its window.

How Netdata helps

  • Certificate countdown as a first-class chart. Netdata collects traefik_tls_certs_not_after and lets you alert on time-to-expiry, so the renewal window itself becomes the alerting threshold instead of browser complaints.
  • Per-second correlation with entrypoint traffic. When a cert expires, correlating the expiry timestamp with the HTTPS entrypoint request rate shows the exact moment client impact started and confirms recovery after the fix.
  • Log and metric correlation on one host. Netdata’s per-host view puts Traefik metrics next to system state (restarts, disk, FD usage), which matters for causes like a volume remount resetting acme.json permissions or frequent restarts driving rate limits.
  • Restart visibility. process_start_time_seconds alongside cert metrics reveals the “Traefik restarts, re-requests certs, hits rate limits” loop that quietly produces this failure.
  • Anomaly detection on the expiry series. A not_after value that keeps approaching now without ever jumping forward is the pattern ML-based anomaly detection flags, without hand-tuning a per-domain threshold.