write_secondary_count is a per-output counter in Fluentd’s monitor_agent API. A nonzero value means the primary output plugin exhausted its retries for at least one chunk, and Fluentd wrote that chunk to the configured <secondary> backup destination instead. The primary pipeline for that output is broken.

This is a ticket-level signal. Data is not lost yet (that is the point of the secondary), but it is no longer flowing where downstream systems expect it. Dashboards, SIEM rules, and alerts that read from the primary destination are now working from a gap.

Two facts make this counter easy to misread. First, it only exists when a <secondary> section is configured on the output. Without one, exhausted retries discard the chunk outright, with only a log line as evidence. Second, it is a cumulative in-memory counter: it resets on Fluentd restart, and a nonzero value from an incident last week looks identical to one from an incident happening right now. You need the delta, not the value.

What this means

Fluentd’s output path is: staged chunk, queued chunk, flush attempt, and on failure a retry cycle with backoff. Retries are bounded by retry_timeout (default 72h) or retry_max_times if set. When a chunk crosses the secondary threshold, Fluentd hands the chunk to the <secondary> plugin and increments write_secondary_count.

flowchart LR
  A[queued chunk] --> B[flush attempt]
  B -->|success| C[purged]
  B -->|failure| D[retry with backoff]
  D -->|recovers| B
  D -->|retry threshold crossed| E[secondary output]
  E --> F[write_secondary_count +1]
  D -->|no secondary configured| G[chunk discarded: data loss]

The trigger point matters. Fluentd switches a chunk to the secondary when the elapsed retry time exceeds retry_secondary_threshold (a ratio of retry_timeout, default 0.8). With the default 72h retry_timeout, a chunk only falls to the secondary after roughly 57 hours of continuous failure. If your write_secondary_count just moved, the primary has likely been down for a long time, or you have lowered retry_timeout or the threshold deliberately.

Common causes

CauseWhat it looks likeFirst thing to check
Primary destination permanently downretry_count climbing, write_count flat for hours, connection refused or timeout in Fluentd logsCurl the destination endpoint from the Fluentd host
Authentication or TLS failureRetries with 401/403 or handshake errors; starts suddenly after a credential or cert changegrep -iE "(tls|ssl|auth|401|403)" in the Fluentd log
Destination rejecting data (schema, quota)Flushes connect but fail; error logs show 4xx responses or per-document rejectionsRecent error lines mentioning the output plugin
Retry configuration too aggressiveSecondary engages quickly; retry_timeout or retry_secondary_threshold set lowOutput config block for retry parameters
Old incident, counter never resetCounter nonzero but retry_count flat, write_count incrementing normally, buffer healthyCompare counter against last Fluentd restart time

Quick checks

# Read the counter per output plugin
curl -s http://localhost:24220/api/plugins.json | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, secondary: .write_secondary_count, retries: .retry_count, writes: .write_count, queue: .buffer_queue_length}'

A nonzero secondary with flat writes and growing queue is an active incident. A nonzero secondary with writes incrementing and an empty queue is residue from a past failure.

# Check current retry state, not just the counter
curl -s "http://localhost:24220/api/plugins.json?with_retry=true" | \
  jq '.plugins[] | select(.plugin_category=="output") | {id: .plugin_id, retry: .retry}'

If retry.steps is large and retry.next_time is far in the future, backoff has pushed the next attempt minutes or hours out. The pipeline is effectively stalled even though it looks like it is retrying.

# Find the failure reason in Fluentd's own log (adjust path for your package)
grep -E "failed to flush|retry|secondary|could not connect" \
  /var/log/td-agent/td-agent.log | tail -30

Look for the log line where chunks were handed to the secondary; it sits right after the last retry failure for each chunk and usually names the plugin and the underlying exception.

# Confirm whether a secondary is configured and where it writes
grep -A5 "<secondary>" /etc/td-agent/td-agent.conf

# Verify the destination independently
curl -sS -o /dev/null -w "%{http_code}\n" --max-time 5 https://your-destination-endpoint/

How to diagnose it

  1. Establish timing. Note the current write_secondary_count, then re-read it after 60 seconds. An incrementing counter means chunks are falling through now. A static counter means the failover already happened (or happened long ago) and the question is whether the primary has recovered.
  2. Check retry state. Pull the retry object for the affected output. Large retry.steps and a distant retry.next_time tell you recovery will be slow even after the destination is fixed.
  3. Read the error. The counter says the primary failed; the Fluentd log says why. Connection refused, auth errors, and data rejection each have different fixes.
  4. Test the destination directly. From the Fluentd host, verify DNS, TCP connectivity, TLS, and auth independently. This separates “destination is down” from “Fluentd’s view of the destination is broken” (expired cert in config, stale credentials).
  5. Locate the secondary data. If the secondary is secondary_file, chunks are being written to the configured directory. Confirm files are actually appearing and growing; a secondary that is itself failing (disk full, permission error) is the worst case, because the fallback is silently not a fallback.
  6. Check buffer headroom. While the primary is down, new events keep arriving. Watch buffer_available_buffer_space_ratios and buffer_queue_length so the ongoing outage does not turn into an overflow on top of the failover.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
write_secondary_count (delta)Chunks falling through to backup right nowAny increment
retry_count and retry objectPrimary is failing; backoff state predicts recovery speedSteps climbing, next_time far out
write_count (delta)Successful primary deliveriesFlat while input continues
rollback_countChunks put back in queue after failed flushesSustained increments
buffer_queue_lengthBacklog depth while the primary is downSustained growth
buffer_available_buffer_space_ratiosTime until overflow during the outageBelow 20% and shrinking
buffer_oldest_timekeyAge of the oldest undelivered dataHours or days behind now

Fixes

Destination down or unreachable

Fix the destination, then be patient or intervene. With exponential backoff, the next retry may be far away even after the destination recovers. Restarting Fluentd resets retry state (and the counter), at the cost of a brief pipeline pause and, with file buffers, a replay burst. That is usually acceptable to drain a long-stalled output quickly.

Authentication or TLS failure

Rotate the credential or certificate into the Fluentd config and reload. A single auth error line may be followed by generic retry messages, so grep broadly. After the fix, watch for write_count resuming and retry clearing.

Destination rejecting data

Schema conflicts and quota rejections do not heal themselves. Stop the flow of offending records (filter or re-route), fix the mapping or quota at the destination, then let retries drain. Do not just raise retry limits: the chunks will keep failing and eventually all land in the secondary.

Re-ingesting secondary data

Chunks in the secondary are not automatically replayed. For secondary_file, the standard recovery path is to read those files back with a separate input (for example in_tail with the matching parser) after the primary is healthy, then remove the files. Plan this before you need it: verify the secondary file format round-trips through your parser.

Retry tuning went wrong

If the secondary engaged too fast, revisit retry_timeout and retry_secondary_threshold. A low threshold trades “fast failover” for “failover during every transient blip,” which scatters data across two destinations constantly.

Prevention

  • Always configure a <secondary> for outputs where data loss is unacceptable. Without it, exhausted retries discard chunks with only a log line. secondary_file ships with Fluentd core; note it only works inside <secondary>, and only buffered outputs support secondary at all.
  • Size the secondary destination. A local directory on the same disk as the buffer is common, but it shares fate with buffer disk pressure. Know its capacity and monitor its growth.
  • Alert on the delta of write_secondary_count, not the value. The counter is cumulative and resets on restart. Any increment should page a human during business hours at minimum.
  • Set retry parameters deliberately. Decide how long the primary may fail before failover, and set retry_timeout and retry_secondary_threshold to match. The defaults imply about 57 hours before the secondary engages.
  • Monitor the leading signals. retry_count, flat write_count, and growing buffer_queue_length all fire long before the secondary threshold is crossed. See the monitoring checklist for the full set.

How Netdata helps

  • Netdata collects Fluentd monitor_agent metrics per output plugin, so write_secondary_count appears alongside retry_count, write_count, and buffer gauges on one timeline instead of separate curl snapshots.
  • The delta view matters here: a rate chart of write_secondary_count distinguishes an active failover (rising slope) from residue of an old incident (flat line at nonzero).
  • Correlating flat write_count, climbing retry_count, and growing buffer_queue_length in one dashboard shows the full arc: primary failing, backlog building, secondary engaging.
  • Anomaly detection on retry and buffer metrics surfaces the primary failure hours before the secondary threshold is crossed, which is when you want to know.
  • Per-plugin breakdowns keep multi-output configurations honest: one output can be failing over to its secondary while others stay healthy, and aggregated metrics would hide that.