$ guides / traefik / traefik-service-server-up-zero ▌

Operations Guides

Traefik service server up at zero: backend health checks are failing

traefik_service_server_up is 0 for one or more URLs in a service. Traefik’s health checker has pulled those backends out of the load balancer rotation. If every URL in the service is at 0, Traefik has nowhere to send requests and is returning 503 to clients right now.

This is a backend-side symptom, not a Traefik-side one. Traefik is doing what you configured it to do: stop sending traffic to servers that fail its checks. The useful question is whether the checks are telling the truth about the backends, and if so, why the backends are failing.

Two things to pin down before fixing anything. First, this metric only exists for services that have health checks configured. If a service has no health check, there is no traefik_service_server_up series for it at all, and absence means unmonitored, not healthy. Second, cross-reference with the 503 rate (traefik_service_requests_total{code="503"}) to confirm whether clients are actually impacted or whether surviving backends are absorbing the load.

What this means

Traefik runs a health checker per backend server. On each interval it probes the backend, and the result drives the gauge: 1 for up, 0 for down. For HTTP services, a backend is healthy if it answers the health check request with a status in the accepted range (2XX-3XX by default). For TCP services, the default check is connection-level: if the TCP connect succeeds, the server is considered healthy. Interval and timeout are configurable per service; the documented defaults are a 30-second interval and a 5-second timeout.

Three operational consequences to keep in mind while diagnosing:

The check may be shallower than your application. A connect-only TCP check, or an HTTP check against a lightweight /healthz that does not touch the database, will report 1 while real traffic paths are broken. Conversely, a check pointed at a heavyweight endpoint can report 0 while the app still serves simple requests.
Failure detection lags by up to one interval. With a 30-second interval, a backend can be dead for nearly 30 seconds while still receiving traffic.
Recovery is instant. When the next check passes, the backend re-enters rotation immediately at full traffic share. There is no ramp-up, so a fragile recovery can re-fail the backend and produce flapping (the gauge alternating 0/1).

Common causes

Cause	What it looks like	First thing to check
Backend process down or port not listening	`server_up` 0, connection refused or timeout on probe, often 502s before the check caught up	Can you TCP-connect to the backend host:port from the Traefik pod/host?
Health check path returns non-2xx/3xx	`server_up` 0, backend reachable and serving real traffic, probe returns 404/401/500	Curl the configured health path directly and look at the status code
App hangs on the health endpoint	`server_up` 0, probe times out rather than errors, real endpoints also slow	Time a manual request to the health path; compare with service latency
Misconfigured health check (wrong path/port/scheme)	`server_up` 0 from the moment the config landed, backend otherwise fine	Recent config change? Compare configured check against what the app actually exposes
Network partition or DNS failure between Traefik and backends	Multiple services lose servers at once, probes fail with connect/resolve errors	Check whether several unrelated services went to 0 simultaneously
Shared dependency failure (database, cache)	All backends of a service fail near-simultaneously, app health endpoint errors	Backend application logs and the dependency’s health
Rolling deployment in progress	`server_up` flapping 0/1 across URLs in sequence, elevated retries	Correlate timing with deploy events

Quick checks

All read-only and safe to run during an incident.

# See which services and URLs are down right now
curl -s http://localhost:8080/metrics | grep traefik_service_server_up

# Confirm client impact: is the service returning 503?
curl -s http://localhost:8080/metrics | grep 'traefik_service_requests_total.*code="503"'

# Check retries, which rise when the pool is shrinking
curl -s http://localhost:8080/metrics | grep traefik_service_retries_total

# Inspect what Traefik's API thinks the service looks like
curl -s http://localhost:8080/api/http/services | head -100

Then probe the backend directly, from the same network position Traefik occupies (exec into the Traefik pod or use a host on the same network):

# Time a request to the configured health path
time curl -sv --max-time 10 http://<backend-host>:<port>/<health-path> -o /dev/null

# If the service is TCP-only, test raw connectivity
nc -zv <backend-host> <port>

The status code and timing from the manual probe usually split the problem in half immediately: connection failure, wrong status, or hang.

How to diagnose it

flowchart TD
  A[server_up = 0] --> B{All URLs down or some?}
  B -->|All| C[Clients getting 503]
  B -->|Some| D[Reduced pool, watch retries and latency]
  C --> E{Manual probe from Traefik network}
  D --> E
  E -->|Connect fails| F[Backend down, port, network, DNS]
  E -->|Wrong status| G[Health path or app-level failure]
  E -->|Hang or timeout| H[App stuck on health endpoint]
  F --> I[Fix backend or connectivity]
  G --> J[Fix path config or the app dependency]
  H --> K[Unstick app, then watch for instant re-entry]

Scope the damage. List every {service, url} pair at 0. If all URLs of one service are down, you are in the 503 scenario. If URLs across multiple unrelated services are down simultaneously, suspect network or DNS between Traefik and the backends rather than the backends themselves.
Confirm client impact. Check the 503 count on the affected service. 503s present means clients see errors now. No 503s means remaining backends are absorbing traffic, but you have lost redundancy and retry load is shifting onto fewer servers.
Probe the health endpoint manually. From Traefik’s network position, request the exact configured health check path, port, and scheme. Three outcomes: connect failure (infrastructure), a status outside 2XX-3XX (note exactly which), or a hang past the check timeout.
Match the failure to a cause. A 404 on the probe means the configured path is wrong for this app. A 401/403 means the check is hitting something that requires auth. A 500 usually means the app’s health endpoint correctly reports a real dependency failure, so go look at the backend’s logs and its database or cache. A timeout means the app is wedged: check its thread pool, connection pool, and downstream calls.
Check what changed. If the metric went to 0 immediately after a config or label change, suspect the health check definition itself (path, port, scheme, hostname) before the application. Traefik silently ignores unknown annotations and labels, so a misspelled key can leave you with a default check that does not match the app.
Watch the recovery. After the backend is fixed, server_up flips back to 1 on the next successful interval and the backend takes full traffic instantly. If the recovery was fragile, expect flapping and elevated traefik_service_retries_total. A flapping backend is often worse for clients than a cleanly down one.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`traefik_service_server_up{service,url}`	Direct health state per backend	Any 0 beyond 2 check intervals; all URLs of a service at 0; repeated 0/1 flapping
`traefik_service_requests_total{code="503"}`	Confirms whether the down pool is user-visible	Sustained 503s on a service with normally healthy backends
`traefik_service_retries_total`	Shrinking pool plus retries amplifies load on survivors	Retry rate rising alongside `server_up` drops
`traefik_service_request_duration_seconds`	Surviving backends under concentrated load slow down	p95 rising as the up-server count falls
`traefik_entrypoint_request_duration_seconds` vs service duration	Separates Traefik overhead from backend time	Both high means the backend is the problem

The most dangerous pattern is the cascade: one backend drops, survivors take its traffic plus retry traffic, they degrade, they fail their checks, and the pool collapses to zero. Declining server_up count plus rising retries is the early warning; the 503 spike is the end state.

Fixes

Backend is actually down or unreachable

Fix the backend, not Traefik. Check the process, the listening port, and the network path from Traefik’s position. In Kubernetes, Traefik’s health checks are independent of kubelet readiness and liveness probes: a pod can pass Kubernetes probes while failing Traefik’s check on a different path or port. If DNS resolution of the backend name is slow or failing, probes fail too, so verify name resolution from the Traefik pod.

Health check path or config is wrong

Point the check at an endpoint the app actually serves, on the right port and scheme, and make sure it does not require authentication Traefik cannot provide. After changing labels or annotations, verify via /api/http/services that the configuration Traefik loaded matches what you intended. Silent annotation rejection means a typo produces no error anywhere; the old or default config just keeps running.

Health check is too shallow (false healthy) or too deep (false down)

If server_up reads 1 while clients get 502s and 504s, your check is shallower than the real traffic path. Make the health endpoint exercise the dependencies that matter (database connectivity, critical downstreams), but keep it cheap enough to answer within the timeout under load. If the check hits a heavyweight endpoint and marks backends down during ordinary load spikes, move it to a lighter path or lengthen the timeout. The accepted status range is configurable if your app legitimately answers with something outside 2XX-3XX.

App hangs on the health endpoint

A hanging health endpoint means the application is wedged, usually on an exhausted connection pool, lock contention, or a dead downstream. Capture logs and thread dumps first, then restart the stuck instance if you must. Make the health endpoint independent of the code paths that wedge. While the app hangs, real requests are likely hanging too, so check traefik_service_request_duration_seconds and 504s alongside.

Pool collapsed and you need traffic flowing

There is no “fail open” option in Traefik: when every backend fails health checks, it will not route to known-bad servers. If the failure is a false negative (check misconfigured, app fine), the fastest safe mitigation is to fix or temporarily relax the health check so backends re-enter rotation. If the failure is real, routing traffic at dead backends would only convert a clean 503 into hangs and timeouts.

Prevention

Configure explicit health checks on every service. A connect-only TCP check misses application-level failure entirely. Without any check, you lose traefik_service_server_up and fly blind on backend health.
Alert on the count, not just all-down. Track how many URLs per service are at 0. Losing one of three backends is a ticket you want during business hours, not a page at 3 a.m. when the last one drops.
Alert on flapping. Repeated 0/1 transitions on a service mean an unstable backend or a badly tuned check, and they feed retry amplification.
Design the health endpoint deliberately. Cheap enough to answer under load, deep enough to catch real dependency failure, and never behind authentication Traefik cannot satisfy.
Plan for instant re-entry. Recovered backends take full traffic immediately, so make sure your app’s warmup (connection pools, caches) tolerates that, or gate real readiness inside the health endpoint itself.
Cross-reference in dashboards. Put server_up, the 503 rate, retries, and service latency for each service on one view. Each signal alone is ambiguous; together they tell the story.

How Netdata helps

Netdata collects traefik_service_server_up per service and URL from Traefik’s Prometheus endpoint, so you can see exactly which backends dropped and when, at per-second resolution during an incident.
Correlating server_up with the per-service 503 rate in the same dashboard answers the first incident question immediately: are clients impacted, or is the pool just smaller?
Overlaying traefik_service_retries_total on the same view exposes the cascade pattern early, when one backend down is still survivable, instead of at full pool collapse.
Service latency next to backend health distinguishes “backends failing checks because they are overloaded” from “checks failing because the check itself is wrong”.
Alerts on all-down and on flapping transitions catch both the acute 503 scenario and the slow-burn unstable backend that metric averages would hide.