$ guides / traefik / traefik-acme-lock-contention ▌

Operations Guides

Traefik ACME lock contention: stuck distributed locks blocking renewal

Every Traefik instance in your HA deployment is healthy. /ping returns 200, traffic flows, backends are up. Yet traefik_tls_certs_not_after keeps creeping toward the current time, and the logs on each instance repeat some variant of “unable to acquire lock”. Certificates are drifting toward expiry and nobody is renewing them.

This is the multi-instance ACME failure mode: Traefik uses a distributed lock in a KV store (Consul or etcd) so that exactly one instance performs ACME issuance and renewal at a time. When the lock holder crashes mid-renewal, is force-killed, or loses its session to a network partition, the lock can be left behind. Every remaining instance refuses to renew because the lock appears held, and the cluster slides toward certificate expiry while looking healthy on every conventional health signal.

This article covers detection, safe recovery, and prevention for deployments that coordinate ACME through a KV store. If you run Traefik v3 Community Edition, read the version note in “What this means” first, because the clustered ACME model changed.

What this means

In a single-instance deployment, Traefik’s ACME resolver stores certificates in acme.json and renews them roughly 30 days before expiry. In a multi-instance deployment, letting every replica renew independently would produce duplicate orders, conflicting writes to shared storage, and Let’s Encrypt rate limit consumption. To prevent that, Traefik elects a single ACME leader through a lock stored in the KV backend. Only the lock holder runs the renewal loop.

The failure is a split-brain by accident. The lock holder dies before releasing the lock. In the v1 KV design the datastore lock has a fixed 20-second TTL, so a holder that disappears should block acquisition only until that TTL expires; if lock state or leadership behaves otherwise, inspect the KV/session directly. Since certificate renewal is the only thing the lock gates, data-plane traffic is unaffected.

flowchart TD
  A[Instance A acquires ACME lock in KV store] --> B[A begins renewal]
  B --> C{A crashes or is partitioned mid-renewal}
  C --> D[Lock key remains in Consul/etcd]
  D --> E[Instances B and C attempt renewal]
  E --> F[Lock appears held - acquisition fails]
  F --> G[Log: unable to acquire lock]
  G --> H[traefik_tls_certs_not_after drifts toward now]
  H --> I[Clients start rejecting expired certs]

Version note. Clustered ACME with a KV-store lock is a Traefik v1-era design; official v3 documentation says the KV approach was dropped in 2.0. Traefik v3 Community Edition does not support shared or clustered ACME at all: each instance manages its own acme.json and its own ACME account, so the stale-lock failure mode does not apply, but neither does coordinated renewal. The officially supported multi-instance ACME path in v3-era deployments is Traefik Enterprise’s distributed ACME agent. If you are on v3 CE with multiple replicas, your equivalent risks are duplicate orders and rate limits, not lock contention. See Traefik ACME rate limit for that failure mode.

Common causes

Cause	What it looks like	First thing to check
Lock holder crashed mid-renewal	Lock key persists in KV store; all instances log lock acquisition failures; one instance restarted recently	`process_start_time_seconds` across replicas; inspect the lock key in Consul/etcd
Force kill or OOM during renewal	Same as above, with an OOM event or SIGKILL in the instance’s history	Container/pod restart reason, kernel OOM log on the node
Network partition between holder and KV store	Holder lost its KV session but kept running; another instance may have taken the lock, or the lock is orphaned	KV connectivity logs on each instance; session/key TTL state in the KV store
Lock TTL longer than cert runway	Lock eventually expires, but not before the renewal window closes	Compare lock age against `traefik_tls_certs_not_after`
Uncoordinated rolling restart	New instance comes up while the old one’s lock is still valid; renewal deferred repeatedly during churn	Correlate lock errors with deployment/rollout timestamps

Quick checks

All read-only. Run these before touching the KV store.

# 1. Confirm the symptom: which certs are closest to expiry
curl -s http://localhost:8080/metrics | grep traefik_tls_certs_not_after

# 2. Confirm all instances are otherwise healthy (the trap: they will be)
curl -s http://localhost:8080/ping

# 3. Look for lock acquisition failures in the logs
#    (patterns seen in the field include "unable to acquire lock"
#    and "Existing key does not match lock use")
grep -iE 'lock|acme' /var/log/traefik/traefik.log | tail -50

# 4. Check for recent restarts that could have orphaned the lock
curl -s http://localhost:8080/metrics | grep process_start_time_seconds

# 5. Inspect the ACME area of the KV store (read-only)
#    The exact lock key path depends on your configured storage prefix.
#    For the v1 datastore, the lock key is exactly the configured ACME KV storage key plus "/lock"; for storage "/traefik/acme/account", inspect "/traefik/acme/account/lock".
consul kv get -recurse /traefik/acme/        # Consul example
etcdctl get --prefix /traefik/acme/          # etcd example

Note on check 5: the key path under your storage prefix varies with how you configured the KV provider and ACME storage. Browse the tree rather than assuming a fixed path. A lock key will typically stand out by name.

How to diagnose it

Establish the timeline. Pull traefik_tls_certs_not_after for the affected CNs and SANs. Let’s Encrypt certificates have a 90-day lifetime and renewal is attempted about 30 days before expiry. If a cert is inside 30 days and has not renewed, renewal is blocked. If it is inside 7 days, renewal has been failing for roughly three weeks and you are in the escalation window.
Confirm the lock story in logs. On each instance, search for ACME and lock-related errors. The distinguishing signature of lock contention, versus challenge failure or rate limiting, is that renewal never starts: instances complain about acquiring the lock, not about challenge responses or ACME server errors. If you instead see challenge errors, go to Traefik ACME challenge failed. If you see rate limit responses, go to Traefik ACME rate limit.
Identify the lock holder. Read the lock key in the KV store. Lock implementations typically record the holder’s identity in the key value. Compare it against your running instances. If the recorded holder no longer exists (terminated pod, decommissioned node), the lock is stale.
Rule out an active holder. Before declaring the lock stale, verify the holder is genuinely gone and not merely partitioned. If the holder is alive and mid-renewal, deleting the lock underneath it lets a second instance start issuing concurrently, which is exactly what the lock exists to prevent. Check whether the holder’s instance ID, pod name, or address appears among live replicas.
Check for a post-deletion race history. If someone already deleted the lock once and instances then logged errors like “Datastore sync error: object lock value: expected X, got Y”, multiple instances raced to re-acquire and the KV state may be inconsistent. The safe recovery from that state is a coordinated restart of all Traefik instances after the lock is cleared, so exactly one instance wins the acquisition cleanly.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`traefik_tls_certs_not_after` (per `cn`, `sans`, `serial`)	The only metric that surfaces this failure; there is no ACME failure metric	Any cert under 30 days not renewing; under 7 days is urgent
`process_start_time_seconds` per replica	A recent restart of one replica is the usual way a lock gets orphaned	One instance restarted while others log lock failures
Traefik logs: lock acquisition errors	The direct evidence; metrics alone only show the consequence	Repeated “unable to acquire lock” across all instances
KV store session/lock key age	Tells you whether the lock is live (session-backed) or orphaned	Lock key present with no corresponding live holder
`traefik_config_last_reload_success` per instance	Rules out config drift as a contributing factor in HA	Divergent timestamps across replicas

Fixes

Clear the stale lock manually

This is the direct fix, and it is disruptive if done carelessly. Deleting a lock that is actually held by a live, mid-renewal instance permits concurrent issuance: duplicate orders, wasted rate limit budget, and potentially conflicting writes to shared certificate storage.

Procedure:

Confirm via the diagnosis steps that the recorded lock holder is gone, not just quiet.
Delete the lock key only, not the whole ACME tree. The account data and issued certificates in the KV store must be preserved.

# DESTRUCTIVE if the holder is still alive. Verify first.
# Replace the key path with the one you confirmed in quick check 5.
consul kv delete /traefik/acme/account/lock     # Consul example
etcdctl del /traefik/acme/account/lock          # etcd example

Watch the logs. One instance should acquire the lock and begin renewal. If several instances log sync errors while racing, restart the Traefik replicas together so acquisition happens cleanly once.

There is no non-disruptive way to revoke a lock. The mitigation for the risk is verification discipline, not a safer command.

Shorten the lock TTL

Check the fixed datastore-lock TTL

In the v1 KV-backed datastore, the transaction lock TTL is fixed in source at 20 seconds and is not exposed as a Traefik configuration option. A crashed holder is bounded by that TTL; do not tune it in Traefik. If a lock remains after the TTL, inspect the KV store and session/lease implementation rather than deleting it blindly.

Tradeoff: a TTL shorter than a worst-case renewal (multiple domains, slow DNS-01 propagation) can expire the lock under a live holder and cause the concurrent-issuance problem from the other direction. Size the TTL to comfortably exceed your slowest observed renewal, not to minimize blocking time.

Designate a single ACME instance

Remove the coordination problem by removing the coordination. Run ACME on exactly one instance (or a small dedicated pair behind health-checked failover) and distribute issued certificates to the other replicas through your own mechanism, or terminate TLS only on the ACME-owning tier.

Tradeoff: you take on certificate distribution yourself, and the ACME instance becomes a renewal single point of failure that needs its own alerting. In exchange, lock contention, duplicate orders, and post-crash races all disappear.

Move off KV-coordinated ACME (v3 CE)

If you are on Traefik v3 Community Edition, clustered ACME is not available, so the fix is architectural: one instance owns ACME per domain set, or use DNS-01 with separate ACME accounts per instance so replicas never collide on orders, or adopt Traefik Enterprise’s distributed ACME agent if coordinated issuance is a hard requirement.

Prevention

Alert on renewal lag, not expiry. A 90-day certificate that has not renewed by day 70 has been failing for ten days. Alert when traefik_tls_certs_not_after crosses 30 days without the serial changing, instead of waiting for the 7-day cliff. Monitor the renewal process, not just the expiry timestamp.
Track restarts against lock errors. A replica restart followed by lock acquisition errors on peers is the canonical precursor. Correlating process_start_time_seconds across replicas with ACME log errors catches this in hours, not weeks.
Make shutdowns clean. OOM kills and force kills during renewal are how locks get orphaned. Give Traefik enough memory headroom and enough shutdown grace to finish or abort an in-flight renewal. On context cancellation the leadership candidate resigns; an in-flight datastore lock remains bounded by its fixed 20-second TTL.
Document the lock key location. During an incident is the wrong time to discover where your storage prefix puts the lock. Record the path and the read/delete commands in your runbook.
Prefer a single ACME owner at small scale. Below the replica count where distributed issuance genuinely pays for itself, one ACME-owning instance plus certificate distribution is operationally cheaper than a distributed lock.

How Netdata helps

Netdata charts traefik_tls_certs_not_after per certificate, so renewal lag is visible as a countdown long before clients start rejecting certificates.
Process uptime per replica makes the “one instance restarted, lock orphaned” correlation visible on a single dashboard instead of across three log tails.
Per-second metric collection catches the restart-and-recover pattern that hourly scrapes miss, which matters when the lock holder flaps.
Because Traefik exposes no ACME failure metric, Netdata’s value here is pairing the expiry gauge with instance restarts and traffic health so you can confirm renewal is the only thing broken.
Alerts on certificates crossing the 30-day renewal window turn a silent three-week drift into a ticket while there is still runway.