Fluentd rollback_count: chunks recycling back into the queue after failed flushes

rollback_count is one of the output-plugin counters exposed by Fluentd’s monitor_agent. It answers a specific question: how many times has a buffer chunk been taken out of the queue for flushing, failed, and been put back? If buffer_queue_length sits stubbornly non-zero while write_count refuses to move, rollback_count tells you chunks are actively cycling through the failure path rather than sitting idle.

This article explains what the counter measures, the mechanism behind it, how it differs from retry_count (the two are easy to conflate), and how to alert on it without paging on noise.

What rollback_count is and why it matters

Every output plugin with buffering maintains a queue of chunks waiting to be flushed. When a flush thread picks up a chunk, that chunk leaves the queue and enters the flushing state. If the write to the destination fails, Fluentd does not discard the chunk: it rolls the chunk back into the queue so a retry can be attempted later. Each rollback increments rollback_count by one.

That makes rollback_count a chunk-level view of delivery failure. Where retry_count counts error events, rollback_count counts physical chunks that had to be re-queued. Both are cumulative counters that reset only on process restart, not when a retry eventually succeeds. A chunk that failed once three days ago and delivered cleanly on the second attempt stays in the counter forever, which directly affects how you alert on it (covered below).

The counter matters for two reasons:

• It confirms the failure is at the write path, not upstream. If rollback_count is incrementing, chunks are reaching the output plugin and the destination is rejecting or failing them. The problem is the destination, the network path, or the output plugin configuration, not parsing or routing.
• Divergence from retry_count is diagnostic. A high rollback_count with a comparatively low retry_count can indicate fast transient failures that resolve quickly, or partial writes where the destination accepted some records but not others.

How it works

The chunk lifecycle is the key mental model. A chunk moves from staged (accumulating events) to queued (ready for flush) to flushing (a flush thread is actively writing it to the destination). On success the chunk is purged. On failure Fluentd returns it to the queue, where it waits for the next retry attempt governed by the output’s backoff settings (retry_type, retry_wait, retry_max_interval, and so on).

flowchart LR
  S[staged] --> Q[queued]
  Q --> F[flushing]
  F -->|write succeeds| P[purged]
  F -->|write fails| RB[rollback]
  RB -->|chunk returned to queue| Q
  RB -.->|rollback_count +1| M[monitor_agent]
  F -.->|retry_count +1 on error| M

Two operational details follow from this mechanism:

• Rolled-back chunks are re-queued for retry ahead of untouched chunks. The failed chunk is put back at the head of the queue (takeback_chunk uses unshift on the dequeued list), so the oldest failed data is retried first, rather than waiting behind newer chunks. During a long outage the same chunk can roll back many times, inflating rollback_count while the number of distinct chunks at risk stays small.
• Rollback and retry are separate events. The rollback is the mechanical act of re-queuing the chunk. The retry is the scheduled re-attempt. A rolled-back chunk sits in the queue until its next scheduled attempt, which under exponential backoff can be minutes or hours away. During that window the chunk contributes to buffer_queue_length but is not being written.

One edge case worth knowing: with file-backed buffers, the rollback operation touches the chunk file on disk, and there is a reported failure mode where rollback raises an IOError (for example when seeking within a chunk file fails; see fluentd issue #3089). If you see rollback-related errors in the Fluentd log rather than just destination errors, check chunk sizes and the health of the filesystem holding the buffer directory.

rollback_count vs retry_count

These two counters move together most of the time, which is why operators often track only one. They measure different things, and the gap between them is informative.

How to read the combinations:

• Both incrementing at roughly the same rate. The common case. Every failed flush produces an error and a rolled-back chunk. Straightforward destination failure: down, unreachable, rejecting connections, or failing authentication.
• rollback_count high, retry_count comparatively low. Points toward fast transient failures that resolve before accumulating many error events, or partial writes where the destination accepted part of the payload and rejected the rest. Elasticsearch bulk responses that return HTTP 200 while rejecting individual documents are the classic case of a destination succeeding at the transport level while losing records at the document level. Whether your output plugin surfaces per-document failures as retries depends on the plugin, so treat this divergence as a prompt to read the Fluentd log and the destination’s rejection metrics.
• retry_count incrementing, rollback_count flat. Less common. Suggests errors recorded without chunks completing the rollback cycle. Check the Fluentd log for exceptions raised outside the normal write path.

In all cases the log is the tiebreaker. The counters tell you chunks are failing; this tells you why:

grep -E "failed to flush|retry" /var/log/td-agent/td-agent.log

Adjust the path for fluent-package or your container setup.

Where it shows up in production

Reading the counter. The counter lives in the monitor_agent output for each buffered output plugin:

# Read rollback_count per output plugin (read-only)
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, rollbacks: .rollback_count, retries: .retry_count, writes: .write_count}'

In multi-worker mode each worker typically exposes its own monitor_agent port (24220 for worker 0, incrementing from there), and each worker has independent buffers and independent counters. Sum across workers for a pipeline-level view, but investigate per worker: a single struggling worker is invisible in the aggregate.

Version availability. rollback_count has been part of the monitor_agent output since the v0.14/v1.0 output API, and v1.6.0 fixed a bug where the counter was not incremented on some rollback paths. It is present in every modern Fluentd; a missing field means a very old Fluentd, not a healthy pipeline.

Prometheus exposure. If you scrape Fluentd with fluent-plugin-prometheus, the output monitor exposes the counter as fluentd_output_status_rollback_count, labeled per output plugin. The same cumulative-counter rules apply.

It never resets on success. Because rollback_count is cumulative for the life of the process, a raw threshold (“rollback_count > 0”) will fire forever after a single transient failure weeks ago. Alert on the rate or the increase over a window, never the absolute value. In PromQL that means rate() or increase(); with monitor_agent polling it means computing the delta between scrapes.

A practical alerting shape. A rollback-to-emit ratio normalizes for throughput better than a raw rate. For example, alert when rollbacks exceed roughly 5 percent of emitted chunks over a 5-minute window:

100 * sum by (type, plugin_id)(rate(fluentd_output_status_rollback_count[5m]))
    / sum by (type, plugin_id)(rate(fluentd_output_status_emit_count[5m])) > 5

Tune the percentage to your tolerance; the ratio shape matters more than the exact number.

What a rising rollback rate leads to. Rollbacks are not data loss by themselves. They are delayed delivery. The danger is what happens next: rolled-back chunks accumulate, buffer_queue_length and buffer_total_queued_size grow, buffer_available_buffer_space_ratios drains toward zero, and when the buffer hits total_limit_size the overflow_action decides whether you block inputs, throw exceptions, or start dropping the oldest chunks. rollback_count is the early end of that cascade, which is exactly where you want to catch it.

Signals to watch in production

The correlation pattern that matters: rollback_count rate rising, write_count flat, queue growing, available space shrinking. That sequence tells you the destination has been failing long enough that the buffer is absorbing the backlog, and it gives you the inputs to compute time-to-overflow before the overflow action fires and data loss begins.

How Netdata helps

• Netdata collects the Fluentd monitor_agent counters per output plugin, so rollback_count, retry_count, and write_count are on the same timeline at per-second granularity. Divergence between them is visible immediately rather than after manual curl sessions.
• Buffer gauges (buffer_queue_length, buffer_total_queued_size, buffer_available_buffer_space_ratios) sit alongside the failure counters, so you can see the rollback-to-backlog cascade as one correlated view instead of three separate checks.
• Because Netdata stores rates derived from cumulative counters, the “counter never resets” trap disappears: restarts show up as a counter reset on the chart, and the derived rollback rate stays correct across them.
• Netdata’s ML anomaly detection on the rollback and retry rates flags unusual recycling behavior even when the absolute rate is below static thresholds, which helps catch slow-burn destination degradation.
• Correlating Fluentd’s rollback rate with host-level signals (disk latency on the buffer directory, network errors, CPU) shortens the path from “chunks are recycling” to the layer actually at fault.

Related guides

• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss
• Fluentd emit_error_count: the number-one under-monitored data-loss signal
• Fluentd failed to flush the buffer: the output cannot deliver and retries begin
• How Fluentd actually works in production: a mental model for operators
• Fluentd memory vs file buffer: why the default buffer loses data on restart
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd monitoring checklist: the signals every production log pipeline needs