$ guides / traefik / traefik-access-log-blocking ▌

Operations Guides

Traefik access log blocking: when logging stalls request handling

Latency spikes across every route on a Traefik instance. Backends report nothing wrong. Service-level latency looks normal, but entrypoint latency is elevated. Health checks pass. If you see this combination and access logging is enabled at high verbosity or high request rate, the log writer is a prime suspect.

Traefik’s access log sits in the request path. With the default configuration (bufferingSize: 0), the access log line for a request is written synchronously from the request-handling goroutine before the request fully completes. When log volume is extreme (thousands of requests per second with verbose field selection), or when the write destination stops draining (full filesystem, blocked pipe), the log write stalls and the goroutine handling that request stalls with it. Requests do not fail; they get slow. That distinction is what makes this failure mode easy to miss.

This guide covers how the blocking happens, how to confirm it with metrics, and how to fix it without losing your access logs.

What this means

Traefik handles each connection with goroutines. The access log middleware wraps the request lifecycle: when a response completes, the middleware formats the log line and writes it to the configured output (stdout by default, or the file set by filePath).

Two configurations behave very differently:

Synchronous (default, bufferingSize: 0): the request goroutine performs the write itself. If the write blocks, the goroutine blocks, and the connection it owns stays occupied. Client-visible duration grows even though the backend responded quickly.
Buffered (bufferingSize > 0): log records enter a channel with capacity bufferingSize, and a background writer drains it. A slow destination first absorbs up to that many pending records; once the channel is full, the request goroutine blocks on the send. This can absorb a short burst, but it is bounded and does not create unlimited memory growth.

The full-filesystem variant is the nastier case. When the partition holding the access log file fills up, writes stop completing. In the synchronous path, every request-handling goroutine queues up behind the stalled writer. Connections accumulate on the entrypoints. Meanwhile /ping still returns 200, because with the default addInternals: false the ping handler never touches the access log path. The instance looks healthy to liveness probes while progressively ceasing to serve traffic.

flowchart LR
  C[Client] --> E[Entrypoint]
  E --> R[Router + middlewares]
  R --> S[Backend service]
  S --> R
  R --> L[Access log write]
  L -->|default: synchronous, same goroutine| F[stdout or log file]
  F -->|full disk or blocked pipe| X[Write stalls]
  X -->|goroutine blocked| E
  E -->|connections accumulate, /ping still 200| C

Common causes

Cause	What it looks like	First thing to check
Synchronous writes at high request rate	Entrypoint p95/p99 elevated, service latency normal, worse at traffic peaks	`bufferingSize` in static config (0 means synchronous)
Full log filesystem	Sudden onset of stalls; writes to the log file fail or block; goroutines and open connections climbing	`df -h` on the partition holding the access log
Blocked stdout pipe	Similar to full disk; common when stdout is consumed by a log collector that is down or backpressured	Is the container’s log driver/collector draining?
Overly verbose logging	Very large log lines (many headers captured), high bytes/sec to the log	`accessLog.fields` and `headers` configuration
Health-check log noise	Log volume inflated by `ping@internal` and other internal entries	`addInternals: true` in config

Quick checks

All read-only and safe to run during an incident. Checks 1-4 assume Prometheus metrics are enabled (metrics.prometheus in the static config) and exposed on port 8080, the default traefik entrypoint; adjust to your deployment.

# 1. Compare entrypoint vs service duration: the blocking signature
curl -s http://localhost:8080/metrics | grep traefik_entrypoint_request_duration_seconds
curl -s http://localhost:8080/metrics | grep traefik_service_request_duration_seconds

# 2. Goroutines climbing without matching traffic = blocked handlers
curl -s http://localhost:8080/metrics | grep go_goroutines

# 3. Open connections accumulating on entrypoints
curl -s http://localhost:8080/metrics | grep traefik_open_connections

# 4. Memory growth if buffered logging is backing up
curl -s http://localhost:8080/metrics | grep process_resident_memory_bytes

# 5. Is the log partition full?
df -h /var/log/traefik    # or wherever filePath points

# 6. Is the log file still being written?
ls -l /var/log/traefik/access.log
date                        # compare file mtime with current time

# 7. Confirm /ping is lying to you about health
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/ping

The key observation is check 1: if traefik_entrypoint_request_duration_seconds is elevated but traefik_service_request_duration_seconds for the same traffic is normal, the time is being spent inside Traefik’s own request path, not in backends. Access log blocking is one of the few things that produces that split uniformly across all services at once. Middleware or TLS problems usually affect specific routes or new connections; log blocking affects everything that completes a response.

How to diagnose it

Establish the duration split. Pull p95 or p99 from both duration histograms over the incident window. Entrypoint high, service normal: internal overhead. Both high: backend problem, and this article does not apply. See Traefik 504 Gateway Timeout if the split points at backends.
Check whether the log output is draining. If logging to a file, check its mtime against wall clock and run df -h on its partition. If logging to stdout from a container, check the container runtime’s log driver and any collector (Fluent Bit, Vector, etc.) for backpressure or downtime. A full filesystem produces sudden, severe stalls; a slow consumer produces gradual degradation.
Confirm goroutine and connection accumulation. Rising go_goroutines and rising traefik_open_connections while request rate is flat or declining means handlers are stuck holding connections. In the full-disk case this climbs until the disk is freed.
Check the effective config. Look at your static configuration for accessLog. If bufferingSize is absent, it defaults to 0 and writes are synchronous. Also check fields, fields.headers, and filters: capturing many request/response headers multiplies bytes per line and therefore write pressure.
Rule out lookalikes. Config reload storms (correlate with the rate of traefik_config_reloads_total) and GC pauses (go_gc_duration_seconds) also create entrypoint-only latency, but they come in bursts tied to reloads or allocation, not sustained across every completed request.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`traefik_entrypoint_request_duration_seconds`	Total time including Traefik-internal overhead	Rising while service duration stays flat
`traefik_service_request_duration_seconds`	Backend-side time as seen by Traefik	Normal here while entrypoint is high = internal blocking
`go_goroutines`	Blocked log writes pin handler goroutines	Monotonic growth without traffic growth
`traefik_open_connections`	Stalled handlers hold connections open	Rising while request rate is flat or falling
`process_resident_memory_bytes`	With buffered logging, a blocked destination grows the in-memory buffer	Steady growth correlated with log write stalls
Disk usage of the log partition	Full filesystem is the common trigger for hard stalls	Trend toward 100% on the partition Traefik logs to
`traefik_entrypoint_requests_total`	Throughput collapse is the late-stage symptom	Rate dropping while upstream demand is unchanged

Two caveats: Traefik exposes no metric for access log write failures or buffer depth, and the Duration field inside the access log itself will not show time spent stalled on the write. Diagnosis has to come from the duration split plus the resource signals above.

Fixes

Enable buffered (asynchronous) log writes

Set bufferingSize in the static configuration so log lines go to an internal buffer drained by a background writer:

# static config
accessLog:
  filePath: "/var/log/traefik/access.log"
  bufferingSize: 100

Tradeoffs: a blocked destination now grows memory instead of stalling requests, so pair this with the memory and disk monitoring above. Lines in the buffer are also lost if the process crashes before they flush. Verify the behavior under load rather than assuming the setting is live. Issue traefik/traefik#9990 documents confusion caused by an incorrect “wait until full” description: the maintainer states any positive value enables asynchronous writes, while the value is buffer capacity rather than a batch size.

Log to stdout and let the collector handle it

Writing to stdout (the default when filePath is unset) moves durability and rotation concerns to the container runtime and your log pipeline, which are usually better equipped to absorb bursts. Tradeoff: if your collector stalls, backpressure lands on Traefik’s stdout pipe instead, so the collector’s health becomes part of Traefik’s availability. Monitor it accordingly.

Reduce log volume

Drop per-header capture unless you need it: accessLog.fields.headers entries are the biggest per-line size multiplier.
Use accessLog.filters (status codes, retry attempts, minimum duration) to skip routine successful requests.
Keep addInternals: false unless you specifically need internal entries; with it on, frequent ping@internal health checks inflate log volume.

Less volume means less write pressure per second at any request rate. This is the cheapest fix and often sufficient on its own.

Free and protect the log partition

If the filesystem is full, freeing space unblocks the writer immediately and accumulated connections drain on their own; you should not need to restart Traefik. Longer term: give the access log its own partition or volume so log growth cannot starve anything else, and alert on its usage trend, not just on “disk full” globally.

For rotation: Traefik reopens its log files on SIGUSR1, which is the signal your logrotate configuration should send after moving the file. Without it, Traefik keeps writing to the deleted inode and the space is never reclaimed, which recreates the full-disk stall. This does not work on Windows.

Prevention

Set bufferingSize deliberately. The default of 0 is the risky configuration. Choose a buffer size and accept the memory tradeoff explicitly.
Monitor the duration split. Dashboard traefik_entrypoint_request_duration_seconds and traefik_service_request_duration_seconds side by side. Divergence is an early warning for several internal failure modes, not just logging.
Alert on the log partition’s usage trend, not on host disk overall. A dedicated volume with a trend alert gives you hours of runway.
Track go_goroutines and traefik_open_connections against request rate. Handler accumulation is the shared symptom of every “Traefik looks healthy but is stalling” failure mode.
Do not trust /ping. It returns 200 while request handling is gridlocked, so liveness probes alone will not catch this failure mode.
Load-test with logging on. Access log configuration is often changed after performance testing. Any change to verbosity, format, or destination should be re-tested at peak request rate.

How Netdata helps

Per-second charts of Traefik’s Prometheus metrics make the entrypoint-versus-service duration split visible in one view, which is the fastest way to confirm internal blocking.
Goroutine count, open connections, and process memory are collected alongside Traefik metrics, so the accumulation pattern (goroutines and connections rising while request rate falls) shows up without manual correlation.
Disk space and inode usage per mount point, with trend alarms, catch the log partition filling before writes stall.
ML anomaly detection on the duration histograms flags the “entrypoint high, service normal” divergence even when absolute latency is still under static thresholds.
Container stdout and collector health can be watched on the same dashboard as Traefik, which matters once logging moves to stdout.