$ guides / traefik / traefik-ping-health-check-trap ▌

Operations Guides

Traefik /ping returns 200 while everything is broken: the health-check trap

Traefik’s /ping endpoint answers one narrow question: “Is the Traefik process alive enough to answer this request?” It does not answer “Can Traefik correctly route production traffic?”

That distinction matters because Traefik is both a data-plane proxy and a control-plane configuration reconciler. The process can stay alive while its Docker socket or Kubernetes API watch has failed, its routing table is stale, every backend is failing health checks, or a certificate is approaching expiry. In all of those states, /ping can still return 200 OK.

Treat /ping as a liveness probe, not a health verdict. A useful Traefik health model needs separate signals for process availability, configuration freshness, routing behavior, backend reachability, TLS state, and resource saturation.

What /ping actually answers

The endpoint must be explicitly enabled, commonly through ping: {} in static configuration or the equivalent --ping option. It is usually exposed on the dashboard or management entrypoint, commonly port 8080, rather than on the public 80 and 443 entrypoints.

# Check the ping endpoint on the management entrypoint
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8080/ping

A 200 response tells you that:

The Traefik process exists.
The management entrypoint accepted the connection.
Traefik generated an HTTP response.

It does not tell you that Traefik can accept traffic on public entrypoints, select a valid router, reach a provider, forward to a healthy backend, or serve an unexpired certificate.

During graceful shutdown, ping.terminatingStatusCode can make /ping return a non-200 status, which lets an orchestrator stop sending new work to the process. That still does not make it a proxy readiness check.

Why liveness is not health

Traefik’s request path has several independently failing stages. /ping bypasses almost all of them.

flowchart LR
  P[/ping request/] --> MP[Management entrypoint]
  MP --> ALIVE[Process response]
  T[Client request] --> EP[Public entrypoint]
  EP --> R[Router match]
  R --> MW[Middleware chain]
  MW --> LB[Service load balancer]
  LB --> B[Backend server]
  CFG[Configuration provider] -. supplies routes .-> R
  ACME[Certificate resolver] -. supplies TLS state .-> EP

The /ping path proves only the top branch. A client request depends on the lower branch and on two background systems: the provider watchers that keep routes current and the certificate resolver that keeps TLS credentials valid.

This is why the endpoint is dangerous as the only health input. A load balancer or Kubernetes probe can keep sending traffic to an instance that is alive but operationally wrong.

What a 200 can hide

Hidden condition	What users see	Better signal
Provider desync	New routes return 404; removed backends keep receiving traffic	`traefik_config_last_reload_success`, entrypoint 404 rate
Stale routing table	Requests arrive, but no current router matches	`traefik_entrypoint_requests_total{code="404"}`
Backend pool collapse	Traefik returns 503 for the service	`traefik_service_server_up` and service 503 rate
Backend application failure	502, 503, or 504 depending on the failure	`traefik_service_requests_total{code=~"5.."}`
Retry amplification	Backend load and latency rise while some clients still see success	`traefik_service_retries_total`
Certificate renewal failure	Clients begin rejecting HTTPS as expiry approaches	`traefik_tls_certs_not_after` plus Traefik ACME logs
File descriptor exhaustion	New connections fail abruptly	`process_open_fds / process_max_fds`
Process memory or goroutine growth	Latency degrades before an OOM restart	`go_goroutines`, `process_resident_memory_bytes`

The first row is especially insidious. When a provider loses connectivity, Traefik retains its last-known configuration instead of flushing routes. Existing routes keep working, so ordinary traffic can look normal while every deployment after the disconnection is invisible.

Provider connectivity and configuration freshness

A live process is not necessarily reconciling current configuration. Traefik may be unable to reach the Kubernetes API, Docker socket, Consul, etcd, or another provider while continuing to serve the last successful routing table.

# Inspect configuration reload metrics
curl -s http://localhost:8080/metrics | grep traefik_config_

Watch traefik_config_last_reload_success as a timestamp and traefik_config_reloads_total as activity. If the timestamp stops advancing while the platform is actively deploying or scaling services, suspect provider desync.

Do not alert on age alone for a static file-based deployment. A configuration unchanged for two weeks may be completely healthy. Correlate the timestamp with deployment activity, Kubernetes events, or expected provider churn.

Distinguish between:

No reload attempts: the provider watcher may be disconnected or stopped.
Reload attempts without a newer successful timestamp: updates may be failing or not applied.
Frequent reloads: a busy environment may be causing a rebuild storm and unnecessary CPU pressure.

Traefik v3 does not provide a reliable traefik_config_reloads_failure_total series for this purpose. Infer failure from reload activity, the success timestamp, logs, and provider behavior.

Route correctness is separate from process health

Traefik generates an entrypoint-level 404 when a request matches no router. That is different from a backend returning a 404 after a service was selected.

A rising rate in traefik_entrypoint_requests_total{code="404"} can mean:

A provider update was missed.
An annotation or label was silently ignored.
A router rule is wrong.
Clients are requesting stale hostnames or paths.
External scanners are enumerating paths.

When a route that should exist is missing, compare the intended configuration with what Traefik actually loaded:

# Inspect routers currently loaded by Traefik
curl -s http://localhost:8080/api/http/routers

Keep this API restricted to trusted networks. It exposes routing and backend topology and must not be reachable from the public internet.

Backend health is separate from backend application health

traefik_service_server_up{service, url} reports whether a backend server is passing Traefik’s configured health check. When every URL for a service drops to 0, Traefik has no healthy backend and returns 503.

Two limitations matter.

First, the metric exists only for services with Traefik health checks enabled. If the series is absent, that means “not monitored by Traefik health checks,” not “healthy.”

Second, a health endpoint can pass while real application paths fail. A backend may return 200 from /health while its database dependency is down and /api returns errors. Cross-check server state against service error rates:

traefik_service_server_up == 0 for all service URLs points to backend pool collapse.
traefik_service_requests_total{code="503"} confirms Traefik has nowhere to send requests.
traefik_service_requests_total{code="502"} indicates Traefik could not get a valid backend response.
traefik_service_requests_total{code="504"} indicates the backend exceeded the configured response time.
Rising traefik_service_retries_total can show intermittent failures and traffic amplification before clients consistently see errors.

Retries deserve attention because they can mask user-facing failure while multiplying backend load. A successful final response after two failed attempts is still three attempts against a struggling backend.

TLS state is invisible to /ping

Certificate renewal runs in the background. Traefik can keep serving HTTPS with the current certificate while renewal repeatedly fails. /ping has no visibility into that condition.

# Inspect certificate expiry metrics
curl -s http://localhost:8080/metrics | grep traefik_tls_certs_not_after

traefik_tls_certs_not_after exposes certificate expiry as a Unix timestamp. For Let’s Encrypt certificates, renewal is normally attempted well before expiry. A certificate inside the renewal window is a signal to verify that automation is still working; a certificate close to expiry means renewal has likely been failing for some time.

No Prometheus metric directly explains an ACME failure. Correlate approaching expiry with Traefik logs for renewal errors, DNS provider problems, challenge reachability, storage issues, or rate limiting. For externally visible hostnames, pair the internal metric with a synthetic TLS check that validates the certificate clients actually receive.

Resource exhaustion can bypass /ping

A reverse proxy can fail at the edge even when its management endpoint still answers.

The most abrupt case is file descriptor exhaustion. Traefik uses descriptors for client connections, backend connections, provider connections, logs, and other process I/O. When process_open_fds approaches process_max_fds, new work can fail with little graceful degradation.

# Inspect process file descriptor metrics
curl -s http://localhost:8080/metrics | grep -E 'process_(open|max)_fds'

Memory and goroutines are slower-moving signals. Rising go_goroutines and process_resident_memory_bytes, disconnected from traffic growth, can indicate hung backend connections or a goroutine leak. The system may stay responsive until the process hits a container limit and is killed.

A /ping request requires very little of the proxy pipeline, so it can remain successful close to the point where accepting or proxying real connections begins to fail.

A practical health model

Do not replace /ping with one larger synthetic request and declare the problem solved. Build layered checks, each with a precise meaning.

Process liveness: /ping answers whether the process can respond. Use it for restart or termination decisions, not for full traffic health.
Public entrypoint acceptance: Verify that the real 80 and 443 listeners accept connections. A management port can work while a public entrypoint is unavailable.
Configuration freshness: Track traefik_config_last_reload_success and compare it with expected deployment activity.
Route correctness: Watch entrypoint 404s and inspect /api/http/routers when intended routes disappear.
Backend capacity: Track traefik_service_server_up where health checks are configured, and use service 5xx rates where they are not.
Client-visible behavior: Monitor 502, 503, and 504 separately because they imply different root causes.
TLS validity: Track traefik_tls_certs_not_after and use external TLS probes for production hostnames.
Resource headroom: Track file descriptors, memory, goroutines, CPU, and connection growth.
Synthetic path checks: For a small number of representative routes, send a request through the normal public path and validate the expected response. This tests more of the chain but does not replace per-service metrics.

The result should let you say precisely which layer failed. “Ping is green” does not do that.

Signals to watch in production

Signal	Why it matters	Warning sign
`process_start_time_seconds`	Detects unexpected process restarts	Repeated starts or a scrape target disappearing
`traefik_entrypoint_requests_total`	Shows whether public traffic is arriving	Sustained drop from baseline or rising entrypoint 404s
`traefik_config_last_reload_success`	Shows configuration freshness	Timestamp frozen during active deployments
`traefik_config_reloads_total`	Shows provider-driven change activity	Flat during churn, or excessively frequent reloads
`traefik_service_server_up`	Shows backend health-check state	One or more servers at 0, especially all URLs at 0
`traefik_service_requests_total` by code	Shows backend and proxy-generated response behavior	Sustained 5xx, especially a growing 503 share
`traefik_service_retries_total`	Reveals intermittent failure and amplified load	Retry rate rising with latency or 5xx
`traefik_service_request_duration_seconds`	Shows service latency from Traefik’s perspective	Per-service percentile deviation from baseline
`traefik_tls_certs_not_after`	Shows remaining certificate lifetime	Expiry approaching without successful renewal
`process_open_fds / process_max_fds`	Measures descriptor headroom	High sustained ratio or continuing growth
`go_goroutines`	Reveals connection or goroutine leaks	Growth unrelated to traffic
`process_resident_memory_bytes`	Shows memory pressure before an OOM kill	Persistent growth toward the container limit

Alert thresholds need deployment context. A static file-provider deployment, a low-traffic staging service, and a high-churn Kubernetes ingress controller do not share a definition of abnormal configuration age, latency, or error rate.

Common misuses

Using /ping as the only load-balancer health check. This keeps a stale or backend-isolated instance in rotation.
Checking /ping on the wrong port. The endpoint is usually on the management entrypoint, not the public traffic entrypoints.
Assuming absent health metrics mean healthy backends. traefik_service_server_up is absent when service health checks are not configured.
Aggregating all 5xx responses together. 502, 503, and 504 point to different failure mechanisms.
Alerting only on certificate expiry. By the time expiry is close, renewal may already have failed repeatedly.
Treating config age as universally bad. Static configurations may legitimately go unchanged for long periods; correlate age with expected changes.
Trusting backend health checks without error-rate correlation. A lightweight health path can pass while real requests fail.

How Netdata helps

Netdata can place /ping availability beside public request rates so a green management check is not mistaken for successful traffic delivery.
Correlating traefik_config_last_reload_success with entrypoint 404s helps identify provider desync without waiting for a developer to report a missing route.
Per-service 502, 503, and 504 trends preserve the distinction between invalid backend responses, exhausted backend pools, and timeouts.
Combining traefik_service_server_up, retries, and service latency exposes cascading backend failure before every backend reaches zero.
Certificate expiry, process restarts, file descriptor usage, memory, and goroutine trends provide the process and resource context that /ping omits.
Anomaly views help compare current behavior with each service’s baseline instead of applying one generic threshold to heterogeneous routes.