Traefik ACME challenge failed: HTTP-01, DNS-01, and TLS-ALPN-01 renewal errors

Your certificates are approaching expiry and Traefik’s logs contain lines like “unable to obtain ACME certificate” or “Error renewing certificate”. HTTPS still works for now because Traefik keeps serving the old certificate, but the clock is running: Let’s Encrypt certificates live 90 days, Traefik attempts renewal 30 days before expiry, and if you are seeing renewal errors today, they have likely been failing silently for weeks.

The uncomfortable part: Traefik exposes no Prometheus metric for ACME renewal failures. The only metric-adjacent signal is traefik_tls_certs_not_after counting down toward expiry. The actual failure details exist only in the logs. That means this failure mode is invisible until someone either reads the logs, alerts on certificate expiry, or gets a browser warning.

The fix depends entirely on which challenge type you are using. HTTP-01, TLS-ALPN-01, and DNS-01 fail for completely different reasons, and the log error for one looks nothing like the log error for another.

What this means

Traefik’s ACME resolver is a background process that asks Let’s Encrypt (or another CA) to prove domain control before issuing a certificate. The CA picks the proof mechanism based on what you configured:

• HTTP-01 (httpChallenge): Let’s Encrypt makes an HTTP request to http://<domain>/.well-known/acme-challenge/<token>. Traefik must be reachable on port 80 and must answer that request itself.
• TLS-ALPN-01 (tlsChallenge): Let’s Encrypt opens a TLS connection on port 443 with the ALPN protocol acme-tls/1. Traefik must terminate that handshake itself and present a temporary challenge certificate.
• DNS-01 (dnsChallenge): Traefik uses your DNS provider’s API to publish a TXT record at _acme-challenge.<domain>. Let’s Encrypt queries DNS for it. This is the only challenge type that can issue wildcard certificates.

Each mechanism has exactly one hard dependency: port 80 reachability, port 443 TLS reachability, or a working DNS provider API plus propagation. Break that dependency, and renewal fails with no signal other than a log line and a shrinking expiry window.

flowchart TD
  A[Renewal due: 30 days before expiry] --> B{Configured challenge}
  B -->|httpChallenge| C[LE requests token over port 80]
  B -->|tlsChallenge| D[LE handshake on port 443, ALPN acme-tls/1]
  B -->|dnsChallenge| E[Traefik creates TXT record via DNS API]
  C --> F{Port 80 reaches Traefik?}
  D --> G{TLS handshake reaches Traefik?}
  E --> H{TXT record visible in public DNS?}
  F -->|No| I[Challenge error in logs]
  G -->|No| I
  H -->|No| I
  F -->|Yes| J[Certificate stored in acme.json]
  G -->|Yes| J
  H -->|Yes| J
  I --> K[traefik_tls_certs_not_after keeps counting down]

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# 1. What is actually being served, and when does it expire?
echo | openssl s_client -servername <domain> -connect <traefik-host>:443 2>/dev/null \
  | openssl x509 -noout -dates -issuer

# 2. Check the expiry metric Traefik exposes
curl -s http://localhost:8080/metrics | grep traefik_tls_certs_not_after

# 3. Find the ACME errors in the logs (adjust for your logging setup)
journalctl -u traefik --since "24 hours ago" | grep -iE 'acme|renew'
# or: docker logs traefik --since 24h 2>&1 | grep -iE 'acme|renew'

# 4. HTTP-01: is port 80 reachable at Traefik from the outside?
# Run this from a host OUTSIDE your network:
curl -v http://<domain>/.well-known/acme-challenge/test
# You want a Traefik response (usually 404 or 503 from Traefik itself),
# not a connection timeout and not an answer from a CDN or other proxy.

# 5. TLS-ALPN-01: does the TLS handshake on 443 land on Traefik?
echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null \
  | openssl x509 -noout -subject -issuer
# The presented cert should be one Traefik serves, not your LB's or CDN's.

# 6. DNS-01: can public resolvers see the challenge record?
dig TXT _acme-challenge.<domain> @1.1.1.1 +short
dig TXT _acme-challenge.<domain> @8.8.8.8 +short

# 7. Check acme.json permissions and validity
ls -l /path/to/acme.json        # must be 600
python3 -m json.tool /path/to/acme.json > /dev/null && echo "valid JSON"

How to diagnose it

1. Confirm the symptom is renewal, not serving. Check traefik_tls_certs_not_after for the affected CN/SANs. If the cert is 30-60 days from expiry, renewal is failing but you have runway. If it is under 14 days, renewal has been broken for weeks and this is now urgent.

2. Identify the configured challenge type. Look at your certificate resolver configuration: httpChallenge.entryPoint, tlsChallenge, or dnsChallenge.provider. The challenge type determines the entire diagnostic path. If you are requesting a wildcard (*.example.com), you must be on DNS-01; the other two cannot issue wildcards.

3. Read the actual error. Grep the logs for acme around the renewal attempts. The error string tells you which of the three dependencies broke: a timeout or connection error points at HTTP-01/TLS-ALPN-01 reachability; a provider API error points at DNS-01 credentials; a propagation or TXT mismatch error points at DNS.

4. Test the dependency directly. For HTTP-01, replay an external request to the challenge path (check 4 above) and confirm the response comes from Traefik, not a CDN, WAF, or load balancer in front. For TLS-ALPN-01, confirm the handshake on 443 terminates at Traefik; a TLS-terminating load balancer in front of Traefik makes this challenge type unworkable because the ALPN challenge never reaches it. For DNS-01, create a test TXT record via the same API credentials Traefik uses, then verify it resolves publicly. If the API call fails, it is credentials. If the API call succeeds but resolvers do not see it, it is propagation.

5. Rule out rate limits. Let’s Encrypt enforces 50 certificates per registered domain per week, 5 duplicate certificates per week, and 300 new orders per account per 3 hours. A Traefik instance crash-looping with a broken resolver config can burn through the duplicate limit in an afternoon. If every other check passes, this is the likely cause, and the log error will say so.

6. Check the storage and the CA endpoint. Verify acme.json is valid JSON with 600 permissions, and confirm caServer is not pointing at the Let’s Encrypt staging directory. Staging renewals “succeed” from Traefik’s perspective but produce certificates no browser trusts.

7. In HA deployments, check the lock. If multiple Traefik instances share ACME state via Consul/etcd and one crashed while holding the renewal lock, every other instance is blocked until the lock expires or is cleared.

Metrics and signals to monitor

Fixes

HTTP-01: restore port 80 reachability

Open the firewall or security group rule for port 80 to the Traefik instance and make sure the traffic lands on Traefik’s HTTP entrypoint, not on a CDN, WAF, or another proxy. If something sits in front of Traefik, the challenge request must be forwarded to Traefik unmodified. Let’s Encrypt does follow redirects, so an HTTP-to-HTTPS redirect on the entrypoint does not by itself break HTTP-01, but the redirect target must still be Traefik. After fixing, renewal retries on Traefik’s own schedule; you do not need to restart.

TLS-ALPN-01: give Traefik the handshake

The TLS connection on port 443 must be terminated by Traefik itself, because only Traefik can present the temporary acme-tls/1 challenge certificate. If a load balancer terminates TLS in front of Traefik, either reconfigure it for TLS passthrough or switch to HTTP-01 or DNS-01. In most cloud load balancer setups, switching challenge type is the pragmatic choice.

DNS-01: fix credentials or propagation

• Rotated token: update the API credentials in Traefik’s environment or config and reload. Verify the new token works by creating a test TXT record through the provider API before waiting for Traefik’s next attempt.
• Propagation delay: Traefik verifies the TXT record before telling Let’s Encrypt to check it. If your DNS provider is slow to propagate, increase the verification delay (delayBeforeCheck in Traefik v2; the dnsChallenge.propagation options in v3).
• Wildcard plus apex: remember DNS-01 is mandatory for wildcards. If only your wildcard domains fail while plain domains renew, that asymmetry confirms a DNS-01-specific problem.

Rate limits: stop the bleeding, then wait

Fix the underlying cause first so Traefik stops generating failing orders. If you are testing configuration, point caServer at the Let’s Encrypt staging endpoint, which has much higher limits. If you have already hit the production weekly limit, there is no override: you wait for the window to slide. This is why rate-limit failures must never be the first time you notice renewal is broken.

Corrupted acme.json or expired-certificate deadlock

Back up acme.json, remove it, and restart Traefik. This forces fresh issuance for all configured domains. Two cautions: restarting Traefik drops in-flight connections, so do it deliberately; and fresh issuance for many domains at once counts against rate limits, so do not do this repeatedly. Note also the asymmetry from the playbook: mid-operation corruption fails silently (in-memory certs keep being served), while corrupted storage at startup makes Traefik hard-fail. If Traefik refuses to start after a restart, check acme.json first.

Stale HA lock

Clear the orphaned ACME lock key in Consul/etcd manually. Consider shorter lock TTLs so a crashed instance does not block renewals for long.

Prevention

• Alert on expiry with real runway. Ticket at under 30 days (the renewal window opened and failed) and page at under 7 days. Waiting for 7 days as your only alert means renewal has been broken for almost two months before anyone looks.
• Alert on the log lines. Since there is no ACME failure metric, scrape or alert on “unable to obtain ACME certificate” and “Error renewing certificate” in Traefik logs. Two consecutive days of renewal errors should page before the expiry metric moves at all.
• Do all resolver testing against staging. Every config change, new domain pattern, or DNS provider migration should be validated against the staging caServer first. Production rate limits do not forgive experimentation.
• Protect acme.json. Enforce 600 permissions, include it in backups, and monitor it for unexpected modification. It is a single point of failure for every certificate you serve.
• Track DNS provider credential rotation. Put DNS API token expiry/rotation on the same calendar as certificate management. A rotated token breaks DNS-01 silently, and wildcards have no fallback.
• Keep certificate inventory visible. Track CN, SANs, and serial numbers so “the cert never changed” is an observable fact, not something you reconstruct during an incident.

How Netdata helps

Netdata surfaces the exact signals this failure mode hides behind:

• traefik_tls_certs_not_after per CN/SAN, charted as time-to-expiry, so a certificate whose renewal has silently failed shows up as a shrinking countdown weeks before browsers complain.
• Anomaly detection on the expiry curve flags certificates that are not being replaced on the expected 30-day-before-expiry cadence, which is the earliest metric-based hint of ACME trouble.
• Log-based correlation puts the ACME error lines from Traefik’s logs next to the expiry metric and any restart events, so you can see “renewal started failing right after this deploy” instead of discovering it from a browser warning.
• Process restart tracking (process_start_time_seconds) catches crash-looping Traefik instances before they burn through Let’s Encrypt’s duplicate-certificate rate limit.
• Config reload freshness (traefik_config_last_reload_success) helps rule in or out a provider-side config problem when the resolver configuration itself may be stale.

The correlation that matters: expiry countdown dropping, ACME errors in logs, and no certificate serial change. Any two of those three means act now.

Related guides

• Traefik certificate expired: when ACME renewal has been failing silently
• Traefik 404 not found: requests arriving with no matching router
• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• Traefik backend connection pool: keep-alive, MaxIdleConnsPerHost, and reuse
• Traefik cannot assign requested address: ephemeral port exhaustion
• Traefik cascading backend failure: how a partial outage becomes a total one
• Traefik circuit breaker: shedding load from a failing backend
• Traefik CLOSE_WAIT pile-up: backend connections that never close
• Traefik config last reload success: monitoring configuration freshness