Fluentd write_secondary_count: the primary output has failed to its backup

write_secondary_count is a per-output counter in Fluentd’s monitor_agent API (exposed there since v1.19.0). A nonzero value means the primary output plugin exhausted its retries for at least one chunk, and Fluentd wrote that chunk to the configured <secondary> backup destination instead. The primary pipeline for that output is broken.

This is a ticket-level signal. Data is not lost yet (that is the point of the secondary), but it is no longer flowing where downstream systems expect it. Dashboards, SIEM rules, and alerts that read from the primary destination are now working from a gap.

Two facts make this counter easy to misread. First, it only exists when a <secondary> section is configured on the output. Without one, exhausted retries discard the chunk outright, with only a log line as evidence. Second, it is a cumulative in-memory counter: it resets on Fluentd restart, and a nonzero value from an incident last week looks identical to one from an incident happening right now. You need the delta, not the value.

What this means

Fluentd’s output path is: staged chunk, queued chunk, flush attempt, and on failure a retry cycle with backoff. Retries are bounded by retry_timeout (default 72h) or retry_max_times if set. When a chunk crosses the secondary threshold, Fluentd hands the chunk to the <secondary> plugin and increments write_secondary_count.

flowchart LR
  A[queued chunk] --> B[flush attempt]
  B -->|success| C[purged]
  B -->|failure| D[retry with backoff]
  D -->|recovers| B
  D -->|retry threshold crossed| E[secondary output]
  E --> F[write_secondary_count +1]
  D -->|no secondary configured| G[chunk discarded: data loss]

The trigger point matters. Fluentd switches a chunk to the secondary when the elapsed retry time exceeds retry_secondary_threshold (a ratio of retry_timeout, default 0.8). With the default 72h retry_timeout, a chunk only falls to the secondary after roughly 57 hours of continuous failure. If your write_secondary_count just moved, the primary has likely been down for a long time, or you have lowered retry_timeout or the threshold deliberately.

Common causes

Quick checks

# Read the counter per output plugin
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, secondary: .write_secondary_count, retries: .retry_count, writes: .write_count, queue: .buffer_queue_length}'

A nonzero secondary with flat writes and growing queue is an active incident. A nonzero secondary with writes incrementing and an empty queue is residue from a past failure.

# Check current retry state, not just the counter
curl -s "http://localhost:24220/api/plugins.json?with_retry=true" | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retry: .retry}'

If retry.steps is large and retry.next_time is far in the future, backoff has pushed the next attempt minutes or hours out. The pipeline is effectively stalled even though it looks like it is retrying.

# Find the failure reason in Fluentd's own log (adjust path for your package)
grep -E "failed to flush|retry|secondary|could not connect" \
  /var/log/td-agent/td-agent.log | tail -30

Look for the log line where chunks were handed to the secondary; it sits right after the last retry failure for each chunk and usually names the plugin and the underlying exception.

# Confirm whether a secondary is configured and where it writes
grep -A5 "<secondary>" /etc/td-agent/td-agent.conf

# Verify the destination independently
curl -sS -o /dev/null -w "%{http_code}\n" --max-time 5 https://your-destination-endpoint/

How to diagnose it

1. Establish timing. Note the current write_secondary_count, then re-read it after 60 seconds. An incrementing counter means chunks are falling through now. A static counter means the failover already happened (or happened long ago) and the question is whether the primary has recovered.
2. Check retry state. Pull the retry object for the affected output. Large retry.steps and a distant retry.next_time tell you recovery will be slow even after the destination is fixed.
3. Read the error. The counter says the primary failed; the Fluentd log says why. Connection refused, auth errors, and data rejection each have different fixes.
4. Test the destination directly. From the Fluentd host, verify DNS, TCP connectivity, TLS, and auth independently. This separates “destination is down” from “Fluentd’s view of the destination is broken” (expired cert in config, stale credentials).
5. Locate the secondary data. If the secondary is secondary_file, chunks are being written to the configured directory. Confirm files are actually appearing and growing; a secondary that is itself failing (disk full, permission error) is the worst case, because the fallback is silently not a fallback. Note that write_secondary_count increments the moment a chunk is handed to the secondary, before the secondary write completes, so a failing secondary still increments it; watch the secondary’s own retries and error log for that.
6. Check buffer headroom. While the primary is down, new events keep arriving. Watch buffer_available_buffer_space_ratios and buffer_queue_length so the ongoing outage does not turn into an overflow on top of the failover.

Metrics and signals to monitor

Fixes

Destination down or unreachable

Fix the destination, then be patient or intervene. With exponential backoff, the next retry may be far away even after the destination recovers. Restarting Fluentd resets retry state (and the counter), at the cost of a brief pipeline pause and, with file buffers, a replay burst. That is usually acceptable to drain a long-stalled output quickly.

Authentication or TLS failure

Rotate the credential or certificate into the Fluentd config and reload. A single auth error line may be followed by generic retry messages, so grep broadly. After the fix, watch for write_count resuming and retry clearing.

Destination rejecting data

Schema conflicts and quota rejections do not heal themselves. Stop the flow of offending records (filter or re-route), fix the mapping or quota at the destination, then let retries drain. Do not just raise retry limits: the chunks will keep failing and eventually all land in the secondary.

Re-ingesting secondary data

Chunks in the secondary are not automatically replayed. For secondary_file, the standard recovery path is to read those files back with a separate input (for example in_tail with the matching parser) after the primary is healthy, then remove the files. Plan this before you need it: verify the secondary file format round-trips through your parser.

Retry tuning went wrong

If the secondary engaged too fast, revisit retry_timeout and retry_secondary_threshold. A low threshold trades “fast failover” for “failover during every transient blip,” which scatters data across two destinations constantly.

Prevention

• Always configure a <secondary> for outputs where data loss is unacceptable. Without it, exhausted retries discard chunks with only a log line. secondary_file ships with Fluentd core; note it only works inside <secondary>, and only buffered outputs support secondary at all.
• Size the secondary destination. A local directory on the same disk as the buffer is common, but it shares fate with buffer disk pressure. Know its capacity and monitor its growth.
• Alert on the delta of write_secondary_count, not the value. The counter is cumulative and resets on restart. Any increment should page a human during business hours at minimum.
• Set retry parameters deliberately. Decide how long the primary may fail before failover, and set retry_timeout and retry_secondary_threshold to match. The defaults imply about 57 hours before the secondary engages.
• Monitor the leading signals. retry_count, flat write_count, and growing buffer_queue_length all fire long before the secondary threshold is crossed. See the monitoring checklist for the full set.

How Netdata helps

• Netdata collects Fluentd monitor_agent metrics per output plugin, so write_secondary_count appears alongside retry_count, write_count, and buffer gauges on one timeline instead of separate curl snapshots.
• The delta view matters here: a rate chart of write_secondary_count distinguishes an active failover (rising slope) from residue of an old incident (flat line at nonzero).
• Correlating flat write_count, climbing retry_count, and growing buffer_queue_length in one dashboard shows the full arc: primary failing, backlog building, secondary engaging.
• Anomaly detection on retry and buffer metrics surfaces the primary failure hours before the secondary threshold is crossed, which is when you want to know.
• Per-plugin breakdowns keep multi-output configurations honest: one output can be failing over to its secondary while others stay healthy, and aggregated metrics would hide that.

Related guides

• Fluentd buffer available space low: computing time-to-overflow before it fires
• Fluentd BufferOverflowError: buffer space has too many data
• Fluentd buffer queue length growing: the output cannot keep pace with the input
• Fluentd config reload failed: SIGHUP that partially applies
• Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes
• How Fluentd actually works in production: a mental model for operators
• Fluentd memory vs file buffer: why the default buffer loses data on restart
• Fluentd monitor_agent not responding: a process that is up but hung
• Fluentd monitoring checklist: the signals every production log pipeline needs
• Fluentd monitoring maturity model: from survival to expert
• Fluentd overflow_action: throw_exception, block, and drop_oldest_chunk
• Fluentd plugin load error at startup: LoadError and missing gems