$ guides / traefik / traefik-401-403-spike ▌

Operations Guides

Traefik 401/403 spike: auth failures, ForwardAuth outages, and brute force

Your dashboard shows traefik_service_requests_total{code="401"} climbing steeply, or the entrypoint-level 4xx panel has turned red. A 401/403 surge is one of the most ambiguous signals a reverse proxy produces, because the same symptom maps to three different situations: your auth layer correctly rejecting bad credentials, your auth layer incorrectly rejecting everyone, or an attacker hammering a login endpoint.

The trap: a ForwardAuth service that is up but degraded (session store exhausted, internal rate limit hit, broken ACL rule) rejects all requests with 401/403. To Traefik this looks like “auth working as designed.” To your users it is a full outage. Conversely, a credential-stuffing run produces the same metric shape but demands the opposite response: block, don’t fix.

What this means

Traefik itself does not generate 401/403. Those codes come from somewhere in the request chain: an auth middleware (BasicAuth, DigestAuth, ForwardAuth) short-circuiting the request, or the backend passing its own 401/403 through. There are no per-middleware Prometheus metrics in Traefik, so you cannot directly see “the ForwardAuth middleware rejected N requests.” You infer the rejection point by comparing entrypoint-level and service-level counters and by reading access logs.

The three scenarios that matter:

Protection working. An auth middleware rejects requests with no valid credentials. Normal background noise, or elevated during an attack.
Auth layer broken. The ForwardAuth service is reachable but returning 401/403 for everyone (degraded session store, misconfiguration), or a credential was rotated and clients still present the old one. Operationally this is an outage wearing an auth costume.
Attack. Credential stuffing or brute force against a login endpoint, or authorization probing (403s) against protected paths. The auth layer is doing its job; the problem is the traffic itself.

Two confounders complicate all three. First, if Traefik sits behind a CDN or cloud load balancer, the connection source IP in access logs is the edge node’s IP, not the real client. The real source is in X-Forwarded-For, which Traefik only logs if you capture that header in the access log config. Second, a rate limiter upstream in the middleware chain returns 429 before the auth middleware runs, which masks the true shape of an attack. Watch 401, 403, and 429 together.

flowchart TD
  A[401/403 spike detected] --> B{Spike on all routers
behind ForwardAuth?}
  B -- Yes, all users affected --> C[ForwardAuth service degraded
or misconfigured ACL]
  B -- No, one endpoint --> D{Single/few source IPs
via X-Forwarded-For?}
  D -- Yes --> E[Brute force /
credential stuffing]
  D -- No, many IPs --> F{Started after a deploy
or key rotation?}
  F -- Yes --> G[Rotated credential /
stale client config]
  F -- No --> H[Authorization probing /
distributed attack]
  C --> I[Check auth service logs
and its dependencies]
  E --> J[Confirm 429s too -
rate limiter may be masking]

Common causes

Cause	What it looks like	First thing to check
ForwardAuth service degraded (session store exhausted, self-rate-limited)	401/403 across all routers using that middleware; valid sessions rejected	Auth service logs and its dependencies (e.g. Redis connection pool)
ForwardAuth misconfiguration (bad ACL, wrong audience)	403 flood starting right after an auth service config change	Recent auth service config diff
Rotated API key / expired client secret	401 spike starting at a known rotation time; affects specific service-to-service traffic	Which clients are failing, and whether they picked up the new credential
Credential stuffing / brute force	401 spike on login endpoints from concentrated source IPs (check X-Forwarded-For)	Access log grouped by real client IP
Authorization probing	403 spike across many paths, often with scanner-style user agents	Access log path and user agent distribution
Rate limiter masking	429s rising while 401s appear lower than expected; attack looks smaller than it is	401 + 403 + 429 rates combined
Errors middleware swallowing BasicAuth challenge	Users report “no password prompt, just an error page”	Middleware chain order on the affected router

One important boundary: if the ForwardAuth service is completely unreachable (connection refused, DNS failure, timeout), Traefik does not return 401. It returns 500. If the client cancels while the auth call is pending, it returns Traefik’s non-standard 499. A pure 401/403 spike means the auth service answered; it just answered “no.”

Quick checks

# 1. Current 401/403 rates from the metrics endpoint
curl -s http://localhost:8080/metrics | grep -E 'code="40[13]"' | grep traefik_service_requests_total
curl -s http://localhost:8080/metrics | grep -E 'code="40[13]"' | grep traefik_entrypoint_requests_total

If 401s appear at the entrypoint level but barely at the service level, the rejection happens in the middleware chain before the request reaches a backend. If both move together, the backend itself is returning the 401s.

# 2. Check 429s too (rate limiter masking the real volume)
curl -s http://localhost:8080/metrics | grep 'code="429"'

# 3. Top source IPs from access logs.
# ClientHost is the connection IP - behind a CDN this is the edge node.
# To group by real client, capture X-Forwarded-For first via
# accessLog.fields.headers.names, then use .request_X-Forwarded-For here.
jq -r '.ClientHost' /var/log/traefik/access.log | sort | uniq -c | sort -rn | head -20

# 4. Which paths are being hit
jq -r 'select(.DownstreamStatus == 401 or .DownstreamStatus == 403) | .RequestPath' \
  /var/log/traefik/access.log | sort | uniq -c | sort -rn | head -20

# 5. Is the ForwardAuth service itself healthy?
curl -s -o /dev/null -w '%{http_code}\n' http://<auth-service>/healthz

# 6. Auth service logs - look for dependency errors, not auth decisions
# (session store timeouts, "unable to save session", pool exhaustion)
kubectl logs -l app=<auth-service> --tail=200 | grep -iE 'error|timeout|redis'

All of these are read-only. Do not restart the auth service or Traefik as a first move: if the cause is a degraded session store, a restart may flush sessions and turn a partial rejection into a total one.

How to diagnose it

Scope the blast radius. Is the spike on every router that uses the auth middleware, or one endpoint? All routers plus all users affected points at the auth layer. One login endpoint points at attack traffic.
Separate middleware 4xx from backend 4xx. Compare traefik_entrypoint_requests_total{code=~"40[13]"} with traefik_service_requests_total{code=~"40[13]"}. Entrypoint-only 4xx means Traefik’s middleware chain rejected the request. Matching 4xx at both levels means the backend generated it.
Check the ForwardAuth service as a dependency, not as an auth oracle. A known real-world pattern: under load, Authelia exhausts its Redis connection pool and returns 401 even for fully valid sessions, with “unable to save updated user session” and Redis timeout errors in its logs. The proxy metric says “auth failures”; the auth service log says “my session store is down.” Always read the auth service’s own logs before concluding anything about credentials.
Identify the real client. Behind a CDN or cloud LB, group access logs by the first X-Forwarded-For entry, not the connection IP. A concentrated source (one IP, one ASN, one user agent, high rate) is brute force. A diffuse source with valid-looking session cookies is a broken auth layer.
Unmask rate limiting. If a RateLimit middleware sits before auth in the chain, the attacker is eating 429s and your 401 graph understates the attack. Sum 401 + 403 + 429 for the true rejection volume. Without a Redis backend, rate limit counters are per Traefik replica, so N replicas with average: 100 allow up to N times that in aggregate, and a distributed attacker can slip under each replica’s bucket.
Check the timeline against change events. Did the spike start at a deploy, an auth service config change, a secret rotation, or a certificate rollover? A step-function onset aligned with a change event is a misconfiguration until proven otherwise. A ramp with no change event is traffic-driven.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`traefik_entrypoint_requests_total{code=~"40[13]"}`	Rejections at the edge, including middleware short-circuits	Sustained rate >5x baseline
`traefik_service_requests_total{code=~"40[13]"}`	4xx actually returned by backends	Rising without entrypoint rise (backend-side auth change)
`traefik_entrypoint_requests_total{code="429"}`	Rate limiter activations; masks attack volume	Rising 429 alongside flat 401
Entrypoint vs service 4xx gap	Localizes the rejection to the middleware chain	Large gap during an incident
Auth service health and dependency metrics (external)	The auth service is a single point of failure for every protected route	Session store errors, pool exhaustion, p99 latency climb
Access log real-client-IP concentration	Distinguishes attack from outage	One IP/ASN generating a large share of 401s
`traefik_service_request_duration_seconds`	A degraded-but-slow auth service adds latency before rejecting	Latency rising before the 401 spike

Fixes

ForwardAuth service degraded

Restore the auth service’s dependency first (session store, database, upstream IdP). If the auth service is self-overloaded, shed load at Traefik with a temporary rate limit on the affected routers rather than letting the auth service drown and reject everything. Tradeoff: rate limiting rejects legitimate users too, but it keeps the auth service alive enough to recover.

Set maxResponseBodySize (and maxBodySize when forwardBody is enabled) to a workload-specific bound; both default to -1 (unlimited). An over-limit request body or auth-server response is returned as 401, which looks exactly like an auth failure. There is no universal provider-specific number, so size the limit from your largest legitimate payload.

Rotated credential or misconfiguration

Roll back the auth service config change or re-distribute the new credential to lagging clients. For service-to-service 401s after a key rotation, the fix is on the client side; Traefik is just reporting it. Use the access log to enumerate exactly which callers still present the old credential.

Brute force / credential stuffing

Block at the edge: tighten the RateLimit middleware on the login path, and make sure sourceCriterion.ipStrategy extracts the real client from X-Forwarded-For (via depth or excludedIPs). Without this, behind a CDN you either rate-limit the CDN edge IP (blocking everyone) or nothing at all. For distributed rate limiting across replicas, use the Redis backend; per-replica counters let attackers multiply through your replica count.

If you use fail2ban against Traefik access logs: the default traefik-auth filter regex expects a username in the log entry, and Traefik does not log one on the initial 401 handshake before credentials are sent. The filter’s mode=aggressive matches those username-less 401s.

Errors middleware breaking the auth UX

If BasicAuth and the Errors middleware are chained, Errors intercepts the 401 before the browser sees the challenge, and users get a custom error page instead of a password prompt. Reorder or scope the Errors middleware so it does not swallow 401s on BasicAuth-protected routers. For ForwardAuth, authSigninURL redirects an upstream 401 to the sign-in URL. For the Errors middleware, statusRewrites can map a configured 401 to another status such as 302.

Log volume during an attack

A heavy 401 flood multiplies access log volume, and a blocked log writer can stall request handling (see Traefik access log blocking). During a sustained attack, filter access logs by status code range (for example statusCodes: ["401", "403"] on a dedicated filtered logger) so forensic capture does not become its own availability problem.

Prevention

Monitor the auth service as a critical dependency. Its health, latency, and its own dependencies (session store, IdP) deserve the same alerting rigor as your backends. A ForwardAuth outage is an outage.
Alert on the 401/403 ratio, not just the rate. A spike where 100% of requests to protected routers are rejected is an auth-layer failure. A spike concentrated on one endpoint from few sources is an attack. Different pages, different responders.
Track 429 alongside 401/403 so rate limiting never silently re-shapes your security signal.
Fix client IP attribution everywhere. forwardedHeaders.trustedIPs at the entrypoint and correct ipStrategy on rate limiters. Note trustForwardHeader is deprecated; migrate to forwardedHeaders.trustedIPs.
Baseline normal 4xx per router. Expired tokens and mistyped passwords are constant background. Alert on deviation, not presence.
Rehearse the credential rotation path. Most “mystery 401 spikes” in calm environments are rotations that did not reach every client.

How Netdata helps

Netdata charts traefik_service_requests_total and entrypoint requests broken out by status code, so a 401/403 spike shows up immediately and you can see in one view whether the rejections happen at the edge or at the service.
Per-second granularity catches the onset shape of the spike: a step function (config change, rotation) versus a ramp (attack building up), which is the first fork in the diagnostic tree.
Correlating the 4xx panel with 429s on the same dashboard exposes rate-limiter masking without manual query work.
Because Netdata also monitors the host and containers, you can put the auth service’s resource usage (CPU, memory, connection counts) next to Traefik’s 401 rate and spot a degraded ForwardAuth dependency instead of chasing phantom credential problems.
Anomaly detection on per-code request rates flags a deviation from the learned 401 baseline without you hand-tuning static thresholds per router.