Traefik 404 not found: requests arriving with no matching router

A client reports that your service returns “404 page not found”. The backend is healthy, its logs show zero traffic, and the 404 body is a bare 19-byte plain-text response, not your application’s error page. That 404 did not come from your backend. It came from Traefik itself.

When a request arrives at a Traefik entrypoint and matches no router, Traefik answers it directly with a 404. The request never touches a service, never runs through a middleware chain, and never reaches an upstream. This is a routing-layer problem with a completely different investigation path from a backend-originated 404, and mixing them up is one of the most common time-wasters in Traefik operations.

The signal that distinguishes the two layers is where the 404 is counted: traefik_entrypoint_requests_total{code="404"} for Traefik-generated 404s versus traefik_service_requests_total{code="404"} for backend-generated ones. Everything in this article follows from that distinction.

What this means

Traefik’s request pipeline is: entrypoint accepts the connection, routers evaluate rules (Host, Path, headers, SNI) in priority order, the matching router’s middleware chain runs, and the request is forwarded to a service. If router matching finds nothing, the pipeline ends there. Traefik synthesizes a 404 at the entrypoint and increments traefik_entrypoint_requests_total{code="404"}. Because no service was selected, service-level metrics show nothing for that request.

Traefik returns 404 rather than 503 for unmatched requests because it cannot know whether the missing route is temporary (a config update in flight) or permanent (a wrong hostname). This behavior is stable across v2 and v3.

An entrypoint 404 means one of four things:

1. The route was never created. A misconfigured annotation or label was silently ignored, so no router exists for this hostname or path.
2. The route existed but Traefik’s view is stale. The provider (Kubernetes API, Docker socket, file) is disconnected or lagging, so new routes have not been loaded.
3. The route exists but the request does not match it. Wrong Host header, wrong path prefix, or the request arrived on an entrypoint the router is not bound to.
4. Nothing should match. Internet scanners, stale DNS, cached URLs, health checks from the wrong source. Background noise.

flowchart TD
  R[Request arrives at entrypoint] --> M{Router match?}
  M -->|no match| E404[Traefik returns 404
entrypoint metric only]
  M -->|match| MW[Middleware chain]
  MW --> SVC[Service / backend]
  SVC --> B404[Backend returns 404
service metric only]
  E404 --> Q1[Check /api/http/routers
and config freshness]
  B404 --> Q2[Check application logs
and route config in app]

Common causes

Quick checks

All read-only. Run them against the Traefik instance actually receiving the traffic.

# 1. Confirm the 404 layer: entrypoint-level, not service-level
curl -s http://localhost:8080/metrics | grep 'traefik_entrypoint_requests_total' | grep 'code="404"'
curl -s http://localhost:8080/metrics | grep 'traefik_service_requests_total' | grep 'code="404"'

If the entrypoint counter is rising and no service counter moves, Traefik is generating the 404. Continue below. If a service counter is rising, you have a backend 404 and this article does not apply.

# 2. See what routes Traefik actually loaded
curl -s http://localhost:8080/api/http/routers | jq '.[] | {name, rule, entryPoints, status, service}'

If the route you expect is absent, the problem is route creation (labels, annotations, provider). If it is present, the problem is rule matching.

# 3. Check configuration freshness
curl -s http://localhost:8080/metrics | grep traefik_config_last_reload_success
curl -s http://localhost:8080/metrics | grep traefik_config_reloads_total

A frozen timestamp while deployments are actively happening means provider desync. traefik_config_last_reload_success is a Unix timestamp in Traefik v3; there is no failure counter, so staleness is inferred from the timestamp not advancing.

# 4. Reproduce the request deterministically
curl -sv -H "Host: app.example.com" http://<traefik-host>:<port>/some/path -o /dev/null

Watch the response body. Traefik’s own 404 is the bare “404 page not found” text. An application 404 usually carries your app’s error format. Then check the access log for this request.

# 5. Inspect the access log entry for an unmatched request
# In JSON access logs, an entrypoint 404 shows no backend involvement:
jq 'select(.DownstreamStatus == 404 and .OriginStatus == 0)' /var/log/traefik/access.log | tail -5

OriginStatus: 0 and OriginContentSize: 0 confirm the request never reached a backend. The entry also records which entrypoint, Host header, and path the request used, which is exactly what you need for rule matching below.

How to diagnose it

1. Fix the layer first. Confirm with quick check 1 that this is an entrypoint 404. Do not touch the backend.
2. Enumerate loaded routers. Pull /api/http/routers and search for the hostname and path in question. Three outcomes: the router is absent (step 3), the router is present but its rule or entrypoint list does not match the request (step 4), or the router is present and correct but recently disappeared and reappeared (step 5).
3. Router absent: check route creation. For Docker, verify the container has the correct traefik.* labels and, if --providers.docker.exposedbydefault=false is set, the traefik.enable=true label. For Kubernetes, check the Ingress or IngressRoute annotations for typos; unknown annotations and labels are silently ignored with no error metric. Compare what you wrote against what the API shows.
4. Router present: check matching. Compare the router’s entryPoints against the port the request arrived on. Binding a router to web (port 80) while the client hits websecure (port 443), or the reverse, is the single most common operator mistake here. Then evaluate the rule itself: does the Host header match exactly, does the path prefix match, are header-based rules satisfied? Reproduce with curl and vary one condition at a time.
5. Router present but flaky or recently changed: check provider health. Look at the age of traefik_config_last_reload_success and whether traefik_config_reloads_total is advancing. Check Traefik logs for provider connection errors, RBAC denials, or watch failures. In Kubernetes, verify permissions: kubectl auth can-i --as=system:serviceaccount:<ns>:<sa-name> list ingresses. If a middleware was recently added or renamed, check whether the router references a middleware that does not exist. A missing middleware prevents Traefik from building the router and serves 404 at the entrypoint; this behavior is not specific to one v3 point release (operator reports include v2.3.5 and v3.3.2). The API still lists the router with status: "disabled" and an error entry rather than omitting it. The v3.7.3 migration note is narrower: it covers the BasicAuth middleware with an empty users list.
6. Rule out noise. If the 404s are for hostnames and paths that should never exist (/.env, /wp-admin, random vhosts), you are looking at internet scanning. That is expected background for any public entrypoint. Alert on rate anomalies, not on existence.
7. In HA deployments, check per-instance divergence. Each replica watches the provider independently. If one instance is stale, requests through it 404 while requests through healthy instances succeed, producing intermittent failures behind the load balancer. Compare traefik_config_last_reload_success across replicas, and compare /api/http/routers output between instances if you suspect drift.

Metrics and signals to monitor

Fixes

Stale config: restore the provider connection

Traefik keeps its last-known-good configuration when a provider disconnects, so existing routes keep working while new ones silently fail. Fix the connectivity (Kubernetes API reachability, RBAC, Docker socket mount, Consul health) and the config resyncs on its own; no manual reload is needed. Do not restart Traefik as a first move: the restart does not fix the provider problem, and on startup before the first successful config load Traefik returns 404 for everything.

Missing route: fix the source configuration

Correct the annotation or label at the source and let the provider push the update. For Docker with exposedByDefault=false, add traefik.enable=true. For Kubernetes, fix the annotation prefix or typo on the Ingress/IngressRoute. Verify against /api/http/routers afterward, because silent rejection means nothing will tell you it worked except the router appearing.

Wrong entrypoint or rule: align the router with the traffic

Bind the router to the entrypoint the clients actually use, or add both entrypoints if the service should answer on 80 and 443. If you want plain HTTP to upgrade instead of 404, that is a redirect configuration on the entrypoint, separate from this fix. Tighten Host and Path rules so they match the real traffic shape; overly specific rules fail closed as 404s.

Invalid middleware reference: fix or remove the reference

When a referenced middleware is missing, Traefik disables that router from the active muxer and serves 404; v3.7.3 added a separate change for BasicAuth with an empty users list. Create the missing middleware, fix the name, or remove the reference. In Kubernetes, watch for ordering: if an IngressRoute is applied before the Middleware CR it references, the router is disabled until the middleware exists, producing transient 404s during deployments.

Optional: return 503 instead of 404 for unmatched requests

If your load balancer or clients handle 503 better than 404, Traefik’s FAQ documents a catchall router: priority 1, a rule that matches everything, pointing at a service with an empty server list (loadBalancer.servers: {}). Unmatched requests then get 503. This is a deliberate behavioral change; make sure it does not mask scanning noise as backend failures in your metrics.

Prevention

• Alert on the ratio, not the absolute count. Entrypoint 404 rate above roughly 5% of total entrypoint requests, or a 3x jump from baseline, is worth a ticket. Background scanning makes absolute counts meaningless on public entrypoints.
• Alert on config freshness. (now() - traefik_config_last_reload_success) > threshold combined with known deployment activity catches provider desync, the most insidious cause of entrypoint 404s.
• Verify routes after every onboarding. A one-line check of /api/http/routers after deploying a new service catches silent annotation rejection before users do.
• Separate entrypoint 404s from service 404s on dashboards. They live in different metric families for a reason. Plot them separately so nobody debugs the wrong layer at 3 a.m.
• In HA, compare replicas. Per-instance config freshness comparison catches the stale-replica pattern that causes intermittent, load-balancer-dependent 404s.
• Lock down the API. The same /api/http/routers endpoint that makes this debugging easy is a full map of your infrastructure. Keep it off public networks.

How Netdata helps

• Netdata charts traefik_entrypoint_requests_total by response code per entrypoint, so Traefik-generated 404s are visually separated from the service-level 404s your backends produce. That split is the first diagnostic fork.
• Tracking traefik_config_last_reload_success alongside the entrypoint 404 rate puts the two most correlated signals for this symptom on one screen: rising 404s plus a frozen reload timestamp points straight at provider desync.
• Anomaly detection on the entrypoint 404 rate flags a departure from baseline (a lost route, a desync) without requiring a static threshold, which matters because scanning noise makes absolute thresholds unreliable.
• Per-instance views in multi-replica deployments make the stale-replica pattern visible: one instance’s 404 rate climbing while the others stay flat.
• Alerting on 404-rate anomalies plus config timestamp age gives you a ticket while the blast radius is still “new routes not working” rather than “half the fleet is stale”.

Related guides

• Traefik 502 Bad Gateway: when the backend is unreachable or returns garbage
• Traefik 503 Service Unavailable: no healthy backends left in the pool
• Traefik 504 Gateway Timeout: the backend is alive but too slow
• Traefik 5xx error rate: telling Traefik-generated errors from backend errors
• How Traefik actually works in production: a mental model for operators
• Traefik monitoring checklist: the signals every production edge router needs
• Traefik monitoring maturity model: from survival to expert
• Traefik /ping returns 200 while everything is broken: the health-check trap