Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is

buffer_oldest_timekey is a timestamp: the timekey of the oldest chunk still sitting in an output plugin’s buffer, waiting to be delivered. Subtract it from the current time and you get a number that answers the question operators care about during an incident: how old is the oldest data that has not reached its destination yet?

That number is more honest than the alternatives. Comparing input and output emit_records rates works for steady streams, but it misleads you for time-sliced outputs (which legitimately hold chunks until the slice expires) and for bursty workloads (where rates oscillate and any short window looks alarming). The oldest timekey does not care about rates. It tells you directly how stale the data is.

This guide covers what the metric is, when it exists, how to compute lag from it, what thresholds make sense, and the gotchas that make naive lag alerts fire on healthy systems.

What buffer_oldest_timekey is and why it matters

Fluentd buffers organize events into chunks. With time-based chunking (<buffer time> with a timekey), every chunk carries a timekey: the truncated timestamp of the time slice it belongs to. The buffer tracks the minimum and maximum of these across staged and queued chunks and exposes them through the monitor agent as buffer_oldest_timekey and buffer_newest_timekey on each output plugin.

The operational meaning:

• now - buffer_oldest_timekey is the age of the oldest undelivered data. If this is 90 seconds, your destination is at most 90 seconds behind reality. If it is 6 hours, you have a severe delivery backlog and every dashboard and alert fed by this pipeline is working with stale data.
• buffer_newest_timekey - buffer_oldest_timekey is the span of time your buffered data covers, useful for understanding how much history you would replay (or lose) if something happened to the buffer.

This is a gauge, sampled at query time. It moves forward in steps as old chunks flush and new ones stage.

flowchart LR
  A[Events arrive] --> B[Staged chunk
timekey = slice start]
  B --> C[Queued chunks
each with a timekey]
  C --> D[Flush to destination]
  C -.-> E[buffer_oldest_timekey
= min timekey]
  E --> F[lag = now - oldest_timekey]
  F --> G{lag > threshold?}
  G -->|yes| H[Delivery backlog]
  G -->|no| I[Pipeline on time]

When the metric exists

buffer_oldest_timekey is only present when time-based chunking is configured for that output’s buffer. If the buffer chunks on tag or uses the default chunking strategy, the field is absent. Do not build a monitoring strategy that assumes the field is always there, and do not treat “field missing” as “lag is zero”.

Outputs with a <buffer time> section (or <buffer tag,time>) and a timekey report it; outputs without it do not. If you want this signal on an output that lacks it, the fix is a config change to time-based chunking, which also changes flush behavior. Treat that as a deliberate tuning decision, not a monitoring patch.

Collecting it manually

The metric comes from the monitor agent API (<source> @type monitor_agent </source>, default port 24220, auto-incrementing per worker in multi-worker mode):

# Oldest and newest timekeys per output plugin
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, oldest: .buffer_oldest_timekey, newest: .buffer_newest_timekey, queue: .buffer_queue_length}'

To compute lag in seconds per output:

# Lag = now - oldest timekey, per output plugin
now=$(date +%s)
curl -s http://localhost:24220/api/plugins.json | \
  jq --argjson now "$now" \
    '.plugins[] | select(.plugin_category=="output" and .buffer_oldest_timekey != null) |
     {id: .plugin_id, lag_seconds: ($now - .buffer_oldest_timekey)}'

In multi-worker mode, each worker has its own monitor agent port (24220, 24221, …) and its own independent buffers. Query every worker. An aggregate that hides one worker’s stalled buffer is a false sense of security.

Thresholds that make sense

There is no universal number of seconds. Derive the threshold from how the output is configured to flush:

• Interval-mode outputs (flush on flush_interval): lag should not exceed roughly 2 * flush_interval. One interval to fill and queue the chunk, one to flush it. Beyond that, something is delaying delivery.
• Time-sliced outputs (e.g., S3 with <buffer time>): lag should not exceed timekey + timekey_wait. A chunk for a given slice legitimately waits until the slice closes plus the wait margin. Anything beyond that means the slice closed and the chunk still has not been delivered, which is a real backlog.

Two rules around these thresholds:

1. Alert on sustained breach, not single samples. Flush cycles, retries, and slice boundaries all create momentary lag spikes that resolve on their own.
2. Derive the threshold from the actual output config. Different outputs on the same Fluentd can have wildly different timekey and flush_interval values. A single global threshold will be simultaneously too tight for one output and too loose for another.

Why this beats rate comparison

The common alternative is alerting on input versus output emit_records divergence. That works, with caveats: you need a window of at least max(2 * flush_interval, 10 minutes), you need input_emit_rate > 0 to avoid division by zero, and time-sliced outputs legitimately diverge for the length of a slice. Bursty workloads produce windows where output rate is zero and everything is fine.

buffer_oldest_timekey sidesteps all of that. It does not ask “are rates balanced right now?” It asks “how old is the oldest thing we owe the destination?” A time-sliced output holding a chunk for its full slice shows lag under timekey + timekey_wait and stays quiet. A stalled output shows lag climbing linearly with wall clock, regardless of burst shape.

Keep rate comparison for detecting silent data loss and routing problems, which oldest-timekey cannot see: it tells you nothing about events that never made it into the buffer at all.

Gotchas that make lag checks lie

The field is absent without time chunking. A lag alert that silently stops evaluating because the field disappeared (after a config change, for example) is a monitoring hole. Alert on the field being unexpectedly absent for outputs where you expect it.

Empty queue, stale value. When the queue drains fully, the oldest-timekey value can lag reality: there is no old data, but the last-reported value may not reflect “nothing is waiting”. This matches how Fluentd publishes the metric: Buffer#statistics only emits oldest_timekey when at least one time-keyed chunk exists, so a drained buffer simply omits the field (null), and a lag expression against it reads null or the dashboard’s last cached value. A lag computation that ignores queue state can report a growing “backlog” on a pipeline that has fully caught up. Gate lag alerts on buffer_queue_length > 0 (or buffer_total_queued_size > 0) so an empty buffer can never page you.

Timekeys move in steps, not continuously. The timekey is a slice boundary, not the timestamp of the oldest individual event. With a one-hour timekey, the oldest timekey can sit an hour behind current time on a perfectly healthy pipeline, because the current slice is still open and accumulating. This is exactly why the threshold for time-sliced outputs is timekey + timekey_wait and not a fixed small number. Do not alert on raw lag without normalizing for the configured timekey.

Per-worker divergence. In multi-worker mode, workers do not share buffer state. One worker can have a growing oldest-timekey while the others are current. Evaluate the metric per worker, per output plugin.

Lag without cause. A rising oldest-timekey tells you delivery is late, not why. Correlate before concluding: retry_count climbing points at destination failure, write_count flat points at a stalled output, buffer_queue_length growing points at the output falling behind, and slow_flush_count or rising average flush time points at destination slowness rather than hard failure.

Signals to correlate

The last row is the escalation that matters most. Lag is a freshness problem. Lag plus shrinking available buffer space is a freshness problem that will soon become a data-loss problem when overflow_action fires. Track both.

How Netdata helps

• Netdata’s Fluentd collector scrapes the monitor agent API per worker, so buffer_oldest_timekey, buffer_newest_timekey, queue length, retry and write counters are collected together at per-second resolution rather than via ad-hoc curl.
• Because all buffer metrics share the same plugin/worker dimensions, you can chart lag next to buffer_queue_length and write_count on one dashboard and see immediately whether staleness comes with a stalled output or is just slice-timing noise.
• Netdata’s anomaly detection on lag and queue-length series surfaces outputs whose oldest data is aging faster than their historical pattern, which catches slow drifts that fixed thresholds miss.
• Correlating Fluentd buffer metrics with host-level signals (disk free on the buffer partition, process RSS) in the same view shortens the path from “data is late” to “and here is how long we have before overflow.”

Related guides

• Fluentd broken pipe / connection reset: dropped output connections and LB timeouts
• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd input emit_records stuck at zero: enable_input_metrics on older versions
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin
• How Fluentd actually works in production: a mental model for operators
• Fluentd input emit_records dropped to zero: ingestion has stopped