$ guides / traefik / traefik-health-check-false-positive ▌

Operations Guides

Traefik health checks pass but requests fail: when the probe lies

Every backend for the service shows traefik_service_server_up = 1. The dashboard is green. And clients are getting 502s and 504s on real requests. Traefik is not malfunctioning: it is faithfully reporting the results of the probe you configured. The problem is that the probe and the production traffic path are testing different things.

This failure mode inverts your usual triage instinct. Normally server_up = 1 means “rule out the backend.” Here it means nothing of the sort: the health check answered a question, just not the one your users are asking. Three distinct mechanisms produce this state, and they have different fixes.

What this means

Traefik’s active health checker is a dedicated worker per backend that sends a GET request to a configured path on a fixed interval. Any 2xx or 3xx response (or a response matching a configured expected status) marks the server healthy and keeps it in the load balancer rotation. That is the entire contract. Traefik has no way to know whether the path you configured is representative of the traffic the service actually serves.

The false positive happens when one of these is true:

The probe path is shallower than the real path. The check hits /healthz, which returns 200 from a static handler with no dependencies. Real traffic goes to /api/v1/data, which needs the database, a downstream service, or a warmed cache. The dependency is down, the application endpoint fails, and /healthz keeps answering 200.
The probe interval is longer than the failure. The default active health check interval is 30s. A backend that dies 2 seconds after a successful probe keeps receiving traffic for up to 28 more seconds. During that window Traefik sends requests to a corpse, generating 502s while server_up still reads 1.
Something else answers the probe. A service mesh sidecar or an intermediate proxy sits between Traefik and the application. The sidecar answers the health check successfully while the application behind it is impaired. Traefik is health-checking the sidecar, not the app.

flowchart LR
  T[Traefik health checker] -->|GET /healthz every 30s: 200 OK| SC[Sidecar or app health handler]
  SC -.->|never tested| APP[Real app path /api/v1/data]
  DB[(Database or dependency)] --> APP
  DB -.x.->|down| FAIL[Real requests: 502 / 504]
  T -->|proxied user traffic| APP

The result in metrics: traefik_service_server_up{service=..., url=...} pinned at 1 while traefik_service_requests_total{code=~"5.."} climbs for the same service. That divergence is the signature. Health status and error rate are supposed to move together; when they do not, the probe is lying.

Common causes

Cause	What it looks like	First thing to check
Health check path has no real dependencies	`server_up=1`, 5xx concentrated on specific routes, app logs show dependency errors	Compare what `/healthz` touches vs. what failing routes touch
Backend died between probes (30s default interval)	Short bursts of 502/504 after a backend restart or crash, then recovery as the checker catches up	Correlate 5xx spike timing with backend pod/process restarts
Sidecar or mesh proxy answers the check	`server_up=1` continuously, real path failing, mesh in the request path	Probe the backend through the same path Traefik uses and inspect what responds
Connection-level failure with passing health checks	Sudden 502 onset across all backends on one host, health checks green	TIME_WAIT and ephemeral port usage on the Traefik host
No real health check configured at all	No `traefik_service_server_up` series exists for the service	Check whether the series is present in `/metrics`; absence means unmonitored, not healthy

The last row matters: traefik_service_server_up only exists for services with health checks enabled. If the metric is absent for a service, you have no probe at all, and Traefik will route to dead backends until something else notices.

Quick checks

All read-only. Run from wherever you can reach Traefik’s metrics endpoint and the backends.

# 1. Confirm the divergence: health state vs real error rate
curl -s http://localhost:8080/metrics | grep traefik_service_server_up
curl -s http://localhost:8080/metrics | grep 'traefik_service_requests_total' | grep 'code="5'

You are looking for a service where every url label shows value 1 while the 5xx counter for the same service is incrementing. Take two samples 60 seconds apart on the requests counter to confirm the 5xx rate is active, not historical.

# 2. Exercise the probe path and the real path on the backend directly
curl -s -o /dev/null -w 'healthz: %{http_code} in %{time_total}s\n' http://<backend>:<port>/healthz
curl -s -o /dev/null -w 'real path: %{http_code} in %{time_total}s\n' http://<backend>:<port>/api/v1/data

If the first returns 200 and the second returns 5xx or hangs, you have confirmed the path mismatch in under a minute. Substitute your actual health check path and a representative production route.

# 3. Check the configured health check interval and path
# (via the API, if the dashboard/API is enabled and reachable internally)
curl -s http://localhost:8080/api/http/services | grep -i -A5 healthcheck

A 30s interval with a shallow path explains both the detection gap and the false confidence.

# 4. Rule out connection-level failure on the Traefik host
ss -tn state time-wait | wc -l
ss -s

If health checks pass but new proxied connections fail, ephemeral port exhaustion or connection pool degradation can produce 502s with green health: the probes may succeed over fresh short-lived connections while the pool for real traffic is starved. A TIME_WAIT count approaching the ephemeral port range is the tell.

# 5. Check retries masking backend instability
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total

A rising retry rate alongside green health checks means backends are failing intermittently between probes, and retries are absorbing some of the damage while amplifying load.

How to diagnose it

Establish the divergence. Pull traefik_service_server_up and the 5xx breakdown from traefik_service_requests_total for the affected service. If all URLs are 1 and 5xx is rising, proceed. If any URL is 0, this is a normal backend failure, not a false positive.
Classify the 5xx. 502 means Traefik connected and got garbage or a reset (backend crashed mid-response, protocol error, connection-level failure). 504 means the backend accepted the request but did not respond within the timeout (alive but stuck, typically on a dependency). 503 with all servers up should not come from the active health checker; if you see it, look for Traefik’s separately configured passive health check; a circuit breaker returns its configured response (503 by default) but does not change server_up, so distinguish those two mechanisms.
Replay both paths against the backend. Run the curl pair from the quick checks. This splits cause 1 (shallow probe path) from everything else.
Check timing. If the 5xx episodes are short (seconds to tens of seconds) and correlate with backend restarts, crashes, or deploys, you are looking at the interval gap. Plot the 5xx spikes against backend restart events.
Inspect what actually answers the probe. If a mesh sidecar is in the path, determine whether the health check response is generated by the sidecar or proxied through to the application. A sidecar answering locally defeats any application-level probe. The same applies to any intermediate proxy between Traefik and the app.
Rule out Traefik-side resource causes. If the failure pattern is sudden-onset 502s across multiple backends at once with green health, check FD usage (process_open_fds / process_max_fds) and TIME_WAIT accumulation before blaming the application.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`traefik_service_server_up`	The health check verdict per backend	All 1s while the rows below degrade
`traefik_service_requests_total{code=~"5.."}`	What backends actually return to real traffic	Rising rate for a fully “up” service: the defining signature
`traefik_service_request_duration_seconds`	Latency climbs before hard failures when a dependency degrades	p95 rising on an all-green service
`traefik_service_retries_total`	Intermittent failures being masked by retries	Retry rate above ~5% of request rate
`process_open_fds / process_max_fds`	502s with green health can be Traefik-side FD pressure	Ratio above 80%
TIME_WAIT socket count (OS level)	Connection churn can fail new backend connections while probes pass	Count approaching the ephemeral port range

Operational rule, worth making a dashboard panel: never render server_up without the 5xx rate for the same service next to it. Either one alone is misleading.

Fixes

Align the probe with real dependencies

Make the health check endpoint exercise the same critical dependencies as production traffic. If /api/v1/data needs the database, the health endpoint should verify database connectivity, not return a static 200. Tradeoff: deep health checks can cause cascading removals when a shared dependency blips, ejecting every backend at once. A common compromise is a check that verifies the dependency with a short timeout and a cheap query, combined with a failure threshold long enough that a single slow probe does not eject the server. This is application-side work; Traefik only consumes the answer.

Shorten the detection gap

Reduce the active health check interval from the 30s default to something in the 5-10s range for services where fast ejection matters, and set the timeout well below the interval. Tradeoff: probe traffic and backend load scale linearly with frequency, and overly aggressive checks on a fragile dependency cause flapping (the recovered backend re-enters rotation instantly at full traffic share, and can fail again immediately). Set the timeout below the interval; Traefik does not validate their interaction, and a slow probe can miss or delay ticker firings, so verify tight timing on your deployed version.

Health check through the mesh, not to it

Where a sidecar answers probes locally, configure the probe so the response genuinely reflects the application: either have the sidecar proxy the check through to the app’s own health endpoint, or point Traefik’s check at an application port that bypasses sidecar-local answering where the mesh design allows it. The specifics are mesh-dependent; the invariant is that the 200 must originate from the application, not the proxy in front of it.

Tune passive checks deliberately

If you rely on failure-based removal driven by real traffic, know that aggressive defaults catch real failures fast but flap on noisy backends. Set passiveHealthCheck.failureWindow (default 10s) and passiveHealthCheck.maxFailedAttempts (default 1) to match how bursty your backends are, and remember that passive observation only sees real traffic, so an idle service never gets probed.

Fix Traefik-side connection pressure

If the diagnosis pointed at FD or ephemeral port exhaustion rather than the application, raise the FD limit (a container default of 1024 is not viable for an edge proxy), verify connection reuse to backends is actually working, and confirm keep-alive timeouts are shorter than any idle timeout of middleboxes between Traefik and the backends.

Prevention

One panel, two series. For every service, chart server_up alongside the 5xx rate. Alert on the divergence, not on either alone: all backends up plus 5xx above baseline is exactly this failure.
Deploy-aware baselines. Brief 5xx spikes after restarts are partly the interval gap and partly backend warmup (connection pools, caches). Tune probe interval and backend readiness together rather than masking both with alert suppression.
Probe design review. When a service is onboarded, review what its health endpoint actually touches. A health endpoint with zero dependencies is a liability, not a safety feature.
Rehearse the sidecar question. In meshed environments, document for each service whether Traefik’s probe reaches the application. Discovering this during an incident costs time you do not have.

How Netdata helps

Netdata surfaces the exact correlation this failure mode hides:

Per-service traefik_service_server_up next to per-service 5xx rate from traefik_service_requests_total, so the “green but failing” divergence is visible without building a custom query.
Per-second granularity on request and error rates, which catches the short 502 bursts caused by the 30s probe interval gap that minute-resolution monitoring averages away.
Retry rate (traefik_service_retries_total) alongside error rate, exposing intermittent backend failure that health checks miss between probes.
Process-level signals on the Traefik host (open FDs vs limit, socket states) for the variant where green health plus 502s is Traefik-side connection pressure, not an application problem.
Latency histograms per service, so dependency degradation shows up as rising p95 on an all-up service before it becomes hard 5xx.